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ABSTRACT 


Littoral  anti-submarine  warfare  (ASW)  operations  generally  focus  on  deterring  and 
eliminating  enemy  diesel-electric  submarines  from  transit  routes  and  protecting  high 
value  units  (HVUs),  such  as  amphibious  warfare  ships  and  logistics  ships.  In  view  of  the 
ASW  challenges  in  the  littorals,  it  is  critical  to  establish  and  maintain  a  highly  effective 
ASW  capability.  The  ASW  techniques  that  we  use  today  are  mostly  effective,  but  it  is 
important  to  explore  new  technologies  and  techniques — such  as  potential  unmanned 
surface  vehicle  (USV)  solutions.  This  study  uses  an  agent-based  simulation  platform 
known  as  Map  Aware  Non-Unifonn  Automata  (MANA)  to  model  the  ASW  effectiveness 
of  USVs  with  the  goal  of  protecting  a  HVU.  The  effectiveness  of  an  ASW  screen 
formation  is  measured  by  the  proportion  of  successful  classifications.  The  results  are 
analyzed  using  comparison  methods,  stepwise  linear  regression,  and  regression  trees.  It  is 
found  from  nearly  390,000  simulated  ASW  missions  that  when  helicopters  are  replaced 
with  USVs,  which  have  the  same  sensor  type  and  capability,  USVs  can  provide  the  same 
classification  effectiveness  in  an  ASW  screen  formation.  The  analysis  also  shows  that  the 
most  significant  characteristic  of  USVs  is  the  classification  range  of  their  dipping  sonar. 
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EXECUTIVE  SUMMARY 


The  Turkish  naval  fleet  conducts  operations  in  its  littoral  waters  to  ensure  free  access  to 
international  waters  and  to  deter  any  threat  to  the  sea  lines  of  communications  (SLOCs). 
Thus,  antisubmarine  warfare  (ASW)  operations  in  Turkish  littoral  waters  generally  focus 
on  deterring  and  eliminating  enemy  diesel-electric  submarines  from  transit  routes  and 
protecting  naval  assets  and  high  value  units  (HVUs),  such  as  amphibious  and  logistics 
ships.  These  operations  enable  naval  forces  to  conduct  more  successful  force  protection 
and  sealift  operations  and  keep  the  SLOCs  open  and  secure. 

Diesel-electric  submarines  are  very  quiet  and  stealthy — and  pose  a  great  threat  to 
Turkey’s  and  allied  forces’  SLOCs.  With  the  increasing  emphasis  on  littoral  ASW,  we 
should  investigate  complementary  abilities  to  address  and  eliminate  diesel-electric 
submarines  with  conventional  forces.  Technological  enhancements  bring  us  new 
capabilities  to  fight  against  stealthy  underwater  threats.  Unmanned  surface  vehicles 
(US  Vs)  have  the  potential  to  enhance  the  current  littoral  ASW  capabilities.  US  Vs  have 
been  used  in  naval  operations  since  World  War  II,  but  recently  these  vehicles  are  gaining 
more  interest  from  modem  navies  with  their  increased  operational  capabilities. 

Effective  employment  and  the  correct  tactical  use  of  USVs  may  offer  a  great  force 
multiplier.  This  can  bring  operational  success,  reduced  risk  and  casualties  to  manned 
platforms,  and  improved  operational  effectiveness.  Based  on  the  discussion  above,  this 
thesis  examines  the  effectiveness  of  unmanned  surface  vehicles  in  anti-submarine  warfare 
with  the  goal  of  protecting  a  high  value  unit. 

This  study  uses  an  agent-based  simulation  platfonn  known  as  Map  Aware  Non- 
Uniform  Automata  (MANA)  to  model  the  ASW  effectiveness  of  USVs  while  considering 
their  advantage  of  long  on-station  time  and  disadvantage  of  low  speed  (as  compared  to 
helicopters).  A  generic  scenario  is  created  to  allow  us  to  experiment  with  potential  USV 
capabilities  in  ASW  missions.  The  modeling  first  focuses  on  building  an  existing  ASW 
screening  scenario  in  MANA.  In  this  scenario,  two  frigates  with  hull-mounted  active 
sonars  are  positioned  on  the  inner  ASW  screen  and  two  ASW  helicopters  with  active 


dipping  sonars  are  positioned  on  the  outer  ASW  screen  to  protect  an  HVU  from 
submarine  attacks.  This  baseline  scenario  provides  a  standardized  benchmark  on  current 
ASW  performance.  The  battlefield  characteristics  and  the  overall  representation  of  the 
baseline  scenario  are  shown  in  the  figure  below.  In  the  first  alternative  scenario,  USVs 
are  included  in  our  model  instead  of  helicopters.  In  doing  so,  USVs  maintain  a  protective 
ASW  barrier  in  front  of  the  surface  group.  This  model  provides  us  some  insights  about 
USVs  as  to  whether  they  can  improve  the  effectiveness  of  ASW  capabilities.  Also,  the 
model  explores  the  overall  effectiveness  of  ASW  screening  when  USVs  are  employed 
with  ASW  helicopters.  The  same  conditions  are  also  explored  for  three  frigate  scenarios. 


(0.0)  * - 140  nm - » 


(20.  8) 


(7. 16) 


(2.  20) 


(7.  24) 


< - 100  nm - ^ 


(random) 


(20.  32) 


The  helicopters  are  stationed  at  the  frigates 
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BATTLEFIELD 


The  battlefield  characteristics  and  the  overall  representation  of  the  baseline 
scenario  (not  drawn  to  scale). 


After  modeling  the  scenarios  in  MANA,  over  390,000  simulated  ASW  screening 
missions  are  executed.  In  designing  our  experiment,  we  apply  a  nearly  orthogonal  Latin 
hypercube  (NOLH)  design  which  provides  good  space-filling  and  statistical  properties. 
We  use  the  experimental  design  to  vary  controllable  and  uncontrollable  factors  and 

xviii 


examine  how  they  affect  the  ability  to  detect  and  classify  a  diesel-electric  submarine 
attempting  to  attack  an  HVU. 

A  comparison  analysis  is  conducted  among  the  scenarios  with  different  numbers 
and  varieties  of  platforms  employed  in  an  ASW  screen  formation.  With  side-by-side  box 
plots  and  one-way  analysis  of  the  means  by  scenarios,  it  was  found  that  when  the 
helicopters  are  replaced  with  USVs,  which  have  the  same  sensor  type  and  capability, 
US  Vs  can  provide  the  same  classification  effectiveness  in  an  ASW  screen  formation.  The 
operating  range  of  the  USVs  is  considerably  shorter  than  the  operating  range  of  the 
helicopters  because  of  the  autonomy  requirements  of  USVs.  Therefore,  USVs  are 
employed  in  the  intermediate  screen  while  the  helicopters  are  employed  on  the  outer 
screen.  This  gives  the  helicopters  a  great  advantage  against  USVs  because  the  helicopters 
can  extend  the  reach  of  the  frigates  to  the  farthest  point  in  the  ASW  screen  and  provide 
an  early  detection  and  classification  of  the  diesel-electric  submarine. 

The  proportion  of  successful  classification  is  used  to  measure  the  effectiveness  of 
ASW  screen  formation  in  a  regression  model.  Based  on  this  measure  of  effectiveness 
(MOE),  the  most  significant  characteristic  of  USVs  is  the  classification  range  of  their 
dipping  sonar.  In  ASW,  the  classification  range  may  depend  on  underwater  conditions, 
background  noise  in  the  ocean,  and  sonar  capability.  The  sonar  parameters  are  mostly 
controllable  because  the  selection  of  the  sonar  type  and  capability  can  be  determined 
during  the  design  process.  But,  the  effectiveness  of  sonar  is  limited  by  environmental 
conditions.  On  the  other  hand,  the  speed  is  viewed  as  an  insignificant  characteristic  of 
USVs  in  the  model  over  the  ranges  explored.  With  this  in  mind,  it  is  important  that  USVs 
self-deploy  to  the  intennediate  screen  ahead  of  the  HVU  with  sufficient  time  and 
endurance  to  satisfy  their  station-keeping  requirements. 

Many  decision  and  noise  factors  have  a  highly  significant  effect  on  the  outcome 
in  our  protective  ASW  scenario.  The  sonar  parameters  of  the  frigates  are  especially 
significant  in  the  model.  The  frigate  sonar  classification  range  has  the  greatest  influence 
on  ASW  mission  success.  The  number  of  frigates  is  another  significant  factor  that  affects 
the  outcome.  Employing  one  more  frigate  in  the  screening  formation,  along  with  its 
assets,  significantly  increases  the  probability  of  detecting  hostile  submarines.  Among  the 


noise  factors,  the  stealthiness  of  the  diesel-electric  submarine  plays  an  important  role  in 
the  model,  as  expected,  since  it  is  a  well-known  crucial  factor  in  littoral  ASW  operations. 
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I.  INTRODUCTION 


"Peace  at  home,  peace  in  the  world.  ” 

-  Mustafa  Kemal  Ataturk 


Since  the  end  of  the  Cold  War,  the  threat  environment  has  shifted  from  open  seas 
to  the  brown  waters,  with  a  greater  emphasis  on  expeditionary  operations,  power 
projection,  and  force  protection  in  littoral  waters.  One  of  the  greatest  military  challenges 
of  today  is  modern  diesel-electric  submarines  operating  in  noisy  and  cluttered  littoral 
environments.  Diesel-electric  submarines  are  very  quiet  and  stealthy — and  pose  a  great 
threat  to  Turkey’s  and  allied  forces’  sea  lines  of  communications  (SLOCs).  With  the 
increasing  emphasis  on  littoral  antisubmarine  warfare  (ASW),  we  should  investigate 
complementary  abilities  to  address  and  eliminate  diesel-electric  submarines  with 
conventional  forces. 

Technological  enhancements  bring  us  new  capabilities  to  fight  against  stealthy 
underwater  threats.  Unmanned  surface  vehicles  (USVs;  see  Figure  1)  have  the  potential 
to  enhance  the  current  littoral  ASW  capabilities  and  reduce  the  risk  to  manned 
platforms  [1].  USVs  have  been  used  in  naval  operations  since  World  War  II,  but  recently 
these  vehicles  are  gaining  more  interest  from  modem  navies  with  their  increased 
operational  capabilities  [2]. 
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Figure  1.  Unmanned  surface  vehicle  (image  from  Textron  Systems, 

http://www.textronsystems.com). 


A.  OVERVIEW 

Over  the  past  two  decades,  the  littoral  waters  have  gained  great  importance.  In 
December  1991,  as  the  world  watched  in  great  surprise,  the  fall  of  the  Soviet  Union  put 
an  end  to  the  Cold  War.  The  post-Cold  War  era  has  had  a  great  effect  on  both  political 
and  military  activities.  This  era  raised  the  possibility  of  unpredictable  regional  wars, 
tensions,  and  conflicts,  especially  in  the  Middle  East,  Southwest  Asia,  Northern  Africa, 
Western  Pacific,  and  Eastern  Europe.  Today,  it  seems  that  in  the  case  of  possible 
conventional  combat,  naval  activities  will  likely  take  place  in  littoral  waters  [3]. 

These  naval  activities  include  force  protection,  surveillance,  littoral  ASW,  mine¬ 
hunting,  mine-clearing,  and  support  for  amphibious  operations.  In  the  littoral  battlespace, 
naval  forces  may  encounter  some  threats  from  potential  enemies  that  are  different  from 
those  in  open  seas  in  the  form  of  quiet  diesel-electric  submarines  (see  Figure  2). 
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Figure  2.  A  diesel-electric  submarine  (image  from  Jane’s  Fighting  Ships, 

https://janes.ihs.com). 


This  unique  platform  is  considered  the  deadliest  threat  in  littoral  waters  because  it 
can  shut  down  its  diesel  engines  and  run  on  a  battery  charge  when  submerged,  resulting 
in  almost  zero  noise,  and  sail  undetected  for  a  long  period  of  time.  Moreover,  high  noise 
and  poor  sound  propagation  conditions  in  the  littoral  waters  give  the  diesel  submarine  an 
even  greater  advantage.  It  can  stay  extremely  quiet  and  submerged  for  up  to  one  week. 
Many  countries  around  the  world  operate  modern  diesel-electric  submarines  because  they 
are  relatively  inexpensive  and  have  greater  effectiveness  in  littoral  waters.  Some  common 
classes  of  modern  diesel-electric  submarines  include  Type  209,  Type  212,  Kilo-class, 
Dolphin-class,  Scorpene,  and  Soryu  [4].  Modem  diesel-electric  submarines  can  be  used 
for  many  purposes,  such  as  threatening  vital  shipping  lanes  and  attacking  high  value  units 
(HVUs)  [5]. 

The  main  role  of  the  Turkish  Navy  is  to  provide  security  for  shipping  lanes  and 
protect  Turkey’s  rights  and  interests  in  its  littoral  waters,  namely  in  the  Aegean,  Eastern 
Mediterranean,  and  the  Black  Sea  (see  Figure  3)  [6]. 
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Figure  3.  Turkey’s  surrounding  seas:  The  Black  Sea,  the  Aegean  Sea,  and  the 
Mediterranean  Sea  (image  from  The  Encyclopedia  of  Earth, 
http://www.eoeearth.org). 


The  Turkish  naval  fleet  conducts  operations  in  its  littoral  waters  to  ensure  free 
access  to  international  waters  and  to  deter  any  threat  to  SLOCs.  Thus,  ASW  operations  in 
Turkish  littoral  waters  generally  focus  on  deterring  and  eliminating  enemy  diesel-electric 
submarines  from  transit  routes  and  protecting  naval  assets  and  high  value  units  (HVUs), 
such  as  amphibious  and  logistics  ships.  These  operations  enable  naval  forces  to  conduct 
more  successful  force  protection  and  sealift  operations  and  keep  the  SLOCs  open  and 
secure. 

Detecting  a  diesel-electric  submarine  is  challenging  and  requires  a  variety  of 
different  platforms  and  sensors.  Each  platform  has  its  own  ASW  capabilities  and  can  be 
employed  in  various  anti-submarine  operations.  To  improve  ASW  effectiveness,  these 
platforms  and  their  sensors  support  each  other  [5].  Due  to  the  operational  challenges  and 
importance  of  littoral  waters,  it  is  critical  to  establish  and  maintain  a  highly  effective 
ASW  capability  [7].  Convoy  or  FIVU  protection  usually  focuses  on  defensive  ASW  and 
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requires  a  detailed  organization  of  escorting  assets.  In  order  to  protect  HVUs  against 
possible  submarine  attacks,  the  Navy  can  employ  surface  warships,  aircraft,  helicopters, 
and  unmanned  underwater  and  surface  vehicles  (UUVs  and  USVs)  equipped  with  active 
or  passive  sonar.  These  ASW  assets  are  deployed  to  patrol  certain  areas  relative  to  the 
HVU’s  position  [8],  Each  type  of  operation  requires  a  certain  number  of  ASW  units, 
manpower,  time,  and  money. 

The  ASW  techniques  that  we  use  today  are  mostly  effective,  but  it  is  important  to 
develop  complementary  skills,  improve  today’s  technology,  and  explore  new  systems, 
such  as  unmanned  solutions.  This  can  increase  the  effectiveness  of  ASW  capabilities  in 
deterring  and  eliminating  enemy  submarines  and  protecting  friendly  forces.  Given 
today’s  increasing  diesel-electric  submarine  threat  from  our  enemies,  it  is  important  that 
the  Navy  has  the  capability  of  operating  USVs  in  naval  operations.  Employing  USVs  in 
ASW  operations  has  the  potential  to  improve  the  efforts  of  existing  ASW  assets. 
Effective  employment  and  the  correct  tactical  use  of  USVs  may  offer  a  great  force 
multiplier.  This  can  bring  us  operational  success,  reduced  risk  and  casualties  to  manned 
platforms,  and  improved  operational  effectiveness  [1]. 

Based  on  the  discussion  above,  this  thesis  examines  the  effectiveness  of 
unmanned  surface  vehicles  in  anti-submarine  warfare  with  the  goal  of  protecting  an 
HVU. 

B.  RESEARCH  QUESTIONS 

The  research  is  guided  by  the  following  questions: 

1.  Can  USVs  give  the  same  effectiveness  as  ASW  helicopters  against  diesel- 
electric  submarines  ahead  of  naval  convoys  or  HVUs? 

2.  What  are  the  main  advantages  and  disadvantages  of  employing  USVs  in 
an  ASW  screen  fonnation? 

3.  Which  characteristics  of  USVs  are  the  most  significant  in  ASW? 

4.  How  do  changes  in  decision  parameters  affect  the  probability  of 
classifying  a  diesel-electric  submarine? 

5.  What  strengths  and  drawbacks  does  the  simulation  software  Map  Aware 
Non-Uniform  Automata  (MAN A)  have  for  modelling  ASW  scenarios? 
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C.  SCOPE  AND  METHODOLOGY 

This  thesis  explores  how  USVs  can  complement  and  extend  existing  ASW 
effectiveness  in  detecting  and  classifying  diesel-electric  submarines.  This  study  also 
addresses  many  controllable  and  uncontrollable  factors  related  to  ASW  to  see  which 
factors  have  the  greatest  effect  on  an  ASW  screen’s  classification  rate.  Results  will  help 
decision-makers  understand  how  USVs  can  be  employed  in  an  ASW  screen  fonnation. 

This  thesis  uses  an  agent-based  simulation  platform  called  MANA  to  model  the 
ASW  effectiveness  of  USVs  while  considering  their  advantage  of  long  on-station  time 
and  disadvantage  of  low  speed  (relative  to  helicopters).  Agent-based  simulation  is  a 
technique  in  which  we  virtually  construct  multiple  autonomous  entities  that  make  their 
own  decisions  and  behave  stochastically  in  their  local  environments  [9]. 

The  modeling  first  focuses  on  building  an  existing  ASW  screening  scenario  in 
MANA.  In  this  scenario,  two  frigates  with  hull-mounted  active  sonars  are  positioned  on 
the  inner  ASW  screen  and  two  ASW  helicopters  with  active  dipping  sonars  are 
positioned  on  the  outer  ASW  screen  to  protect  an  HVU  from  submarine  attacks.  This 
baseline  scenario  provides  us  a  standardized  benchmark.  In  the  first  alternative  scenario, 
USVs  are  included  in  our  model  instead  of  helicopters.  In  doing  so,  USVs  will  maintain  a 
protective  ASW  barrier  in  front  of  the  surface  group.  This  model  provides  us  some 
insights  about  USVs  as  to  whether  they  can  improve  the  effectiveness  of  ASW 
capabilities.  Also,  we  explore  the  overall  effectiveness  of  ASW  screening  when  USVs  are 
employed  with  ASW  helicopters.  The  same  conditions  are  also  explored  for  three  frigate 
scenarios. 

After  modeling  the  scenarios  in  MANA,  nearly  390,000  simulated  ASW  missions 
are  executed.  In  designing  our  experiment,  we  apply  a  nearly  orthogonal  Latin  hypercube 
(NOLH)  design  which  provides  good  space-filling  and  statistical  properties  [10].  We  use 
the  experimental  design  to  vary  controllable  and  uncontrollable  factors  and  examine  how 
they  affect  the  ability  to  detect  and  classify  a  diesel-electric  submarine  attempting  to 
attack  an  HVU. 
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D.  LITERATURE  REVIEW 

A  literature  review  is  conducted  to  examine  previous  studies  and  documents  about 
USV  employment  in  naval  operations.  These  studies  and  documents  do  not  cover  the 
scope  of  this  thesis,  but  the  methodologies  and  insights  utilized  in  these  studies  are 
important  to  review  before  moving  on  to  the  model  development  phase. 

In  her  master’s  thesis,  Steele  (2004)  studies  the  perfonnance  of  a  USV  with 
respect  to  its  current  capabilities  in  information,  surveillance,  and  reconnaissance  (ISR) 
and  force  protection  (FP)  missions  [11].  She  uses  an  agent-based  simulation  platform 
called  PYTHAGORAS  to  build  her  mission  scenarios.  Steele’s  study  explores  alternative 
configurations  of  a  prototype  USV  and  its  operational  use.  The  results  of  the  study 
provide  some  useful  operational  and  tactical  insights — ultimately,  she  recommends  that 
the  U.S.  Navy  use  USVs  in  maritime  missions. 

In  his  thesis,  Abbott  (2008)  examines  the  effective  use  of  an  employed  LCS 
squadron  to  provide  analytic  support  for  the  LCS  program  office  [12].  He  builds  three 
different  scenarios  in  MANA  based  on  the  current  mission  packages  for  LCS:  Anti- 
Surface  Warfare  (ASuW),  Anti-Submarine  Warfare  (ASW),  and  Mine  Warfare  (MIW). 
This  study  touches  on  USVs  in  one  of  these  scenarios.  In  the  ASW  scenario,  a  USV  is 
employed  to  act  similarly  to  an  ASW  helicopter.  It  is  assumed  that  the  USV  has  a  dipping 
sonar  capable  of  finding  a  submerged  submarine.  In  this  model,  once  a  USV  detects  a 
submarine,  it  helps  to  localize  the  submarine  and  passes  this  information  to  an  LCS  for 
prosecution.  With  respect  to  the  ASW  scenario,  the  results  show  that  sensor  systems  play 
a  significant  role. 

In  2013  the  Research  And  Development  (RAND)  Corporation  published  U.S. 
Navy  Employment  Options  for  Unmanned  Surface  Vehicles  (USVs)  with  the  sponsorship 
of  the  Office  of  the  Chief  of  Naval  Operations,  Assessment  Division  (OPNAV  N81)  [13]. 
This  report  researches  the  prospective  suitability  of  USVs  for  U.S.  Navy  missions  and 
functions.  Firstly,  it  introduces  the  current  and  emerging  USV  marketplaces  to 
understand  the  capabilities  of  platforms  for  U.S.  Navy  demands.  Secondly,  it  develops 
concepts  of  employment  to  find  out  how  USVs  could  be  used  in  naval  missions  and 
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functions.  It  then  analyzes  these  concepts  of  employment  to  specify  highly  suitable 
missions  and  functions.  The  report  identities  62  potential  missions  and  functions  for  USV 
employment  and  conducts  a  suitability  analysis  for  these  missions  and  functions  based  on 
pre-dclined  criteria.  The  results  of  this  analysis  show  that  among  the  62  missions  and 
functions,  27  of  them  are  considered  as  highly  suitable  missions  and  functions  for  USV 
employment.  Mostly,  ASW  missions  fall  in  the  category  of  less  suitable  missions  and 
functions,  but  unarmed  ASW  area  sanitization — a  mission  to  detect  and  classify 
adversary  submarines — is  deemed  a  highly  suitable  mission  in  the  emerging  USV  market. 
Unarmed  ASW  area  sanitization  focuses  on  ensuring  that  no  enemy  submarine  is 
operating  on  transit  routes  or  providing  early  warning  when  an  enemy  submarine  is 
detected  and  classified.  In  this  mission,  USVs  are  deployed  to  an  operating  area  ahead  of 
an  HVU  with  sufficient  time  to  search  for  enemy  submarines  before  the  HVU  arrives. 
USVs  may  conduct  this  mission  overtly  or  covertly.  While  overt  ASW  operations  dictate 
the  use  of  active  sonar,  covert  operations  would  use  passive  sonar  for  better  concealment. 
Employing  multi-mission  manned  platforms  for  this  mission  is  expensive,  both 
monetarily  and  in  tenns  of  valuable  resources.  Reducing  the  risk  to  manned  platforms 
and  freeing  them  for  other  missions  are  the  main  advantages  of  using  USVs  for  this 
mission. 

E.  THESIS  OUTLINE 

Chapter  II  summarizes  basic  concepts  of  ASW,  informs  the  reader  about  USVs 
currently  employed  by  the  U.S.  Navy,  and  discusses  the  agent-based  modeling  and 
simulation  modeling  software  MANA.  Chapter  III  explains  model  development  and 
describes  each  scenario  used  in  this  thesis.  Modeling  assumptions  and  limitations  are 
covered  as  well  as  agent  descriptions.  Chapter  IV  discusses  the  exploration  of  the  model. 
At  the  beginning  of  this  chapter,  we  describe  the  design  of  experiment  (DOE)  techniques 
that  are  used  to  investigate  the  simulation.  Then,  we  explain  all  the  controllable  and 
uncontrollable  factors  that  could  potentially  affect  the  outcome.  After  the  discussion  of 
the  model  exploration,  the  model  output  is  analyzed  using  several  statistical  techniques, 
such  as  least  squares  regression  and  partition  trees.  Following  this,  factor  significance  is 

examined.  Chapter  V  concludes  with  a  summary  of  the  thesis  and  provides  some 
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recommendations  and  useful  insights  for  decision-makers.  It  also  includes  some  ideas  and 
recommendations  for  further  research. 
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II.  BACKGROUND 


“The  maritime  should  be  considered  as  Turkey ’s  major  national  ideal  and 
we  have  to  achieve  it  in  less  time.  ” 

-  Mustafa  Kemal  Ataturk 


This  chapter  provides  a  basic  operational  and  theoretical  background  on  USVs 
and  ASW  to  help  guide  the  development  of  the  models  and  discussions  in  this  thesis. 
Since  this  study  analyzes  an  ASW  scenario,  it  is  important  to  have  some  basic 
information  about  the  concepts  and  components  of  ASW.  We  then  provide  an  overview 
of  technological  developments  of  anti-submarine  warfare  unmanned  surface  vehicles 
(ASW  USVs)  and  introduce  the  major  missions  of  USVs  in  littoral  ASW  operations.  We 
also  provide  some  background  on  agent-based  modeling  and  MAN  A  software. 

A.  ANTI-SUBMARINE  WARFARE 

The  main  purpose  of  ASW  is  to  prevent  our  enemies  from  using  their  submarines 
effectively  [14].  ASW  is  a  branch  of  underwater  warfare  that  employs  a  mix  of  naval 
platforms  such  as  surface  warships,  helicopters,  maritime  patrol  aircraft,  and  submarines 
to  detect,  track,  damage,  or  destroy  enemy  submarines.  In  the  near  future,  we  will  have 
the  capability  of  operating  a  variety  of  unmanned  vehicles  in  ASW  operations.  These 
various  ASW  platforms  have  different  system  and  sensor  capabilities. 

In  order  to  understand  the  proposed  model,  it  is  important  to  understand  the  nature 
of  ASW.  We  briefly  describe  littoral  ASW  concepts,  processes,  platforms,  and  the 
acoustic  environment. 

1.  Littoral  ASW  Concept 

In  littoral  waters  the  diesel-electric  submarine  remains  one  of  the  most  effective 
ways  to  threaten  operational  capability.  Curt  Lundgren  addresses  the  submarine  threat  in 
his  article  “Stealth  in  the  Shallows:  Sweden’s  Littoral  Submariners”  published  in  Jane’s 
Navy  International'. 
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In  the  Royal  Swedish  Navy’s  experience,  the  conditions  make  it  very 
difficult  to  detect  and  prosecute  a  submarine.  Put  simply,  the  Baltic  is  an 
ASW  officer’s  nightmare  and  a  submariner’s  heaven.  ...  For  an  aggressor, 
submarines  operating  in  the  littoral  environment  are  very  bad  news,  and 
the  resources  and  time  required  to  find  and  prosecute  a  submarine  threat 
are  likely  to  be  disproportionately  high.  ...  The  well-designed  and 
proficiently  crewed  submarine  remains  a  highly  stealthy  platfonn  in  the 
littoral  environment.  [15] 

Adversaries  may  conduct  underwater  operations  on  transit  routes  to  threaten 
merchant  convoys  and/or  HVUs.  With  the  purpose  of  enabling  joint  or  naval  forces  to 
conduct  more  successful  operations,  littoral  ASW  has  to  focus  on  denying  submarine 
threats  access  to  our  areas  of  interest  and  preparing  more  secure  spaces  for  friendly 
forces.  In  regional  maritime  conflicts,  it  is  important  to  establish  a  clear  battlespace  and 
transit  HVUs  through  the  littoral  waters  [16]. 

In  the  near  future  the  environment  in  the  littoral  waters  will  be  more  complex  and 
chaotic  due  to  higher  density  traffic  and  a  more  cluttered  environment.  Denying  and 
eliminating  stealthy  submarines  will  be  more  difficult  [17].  Because  the  littoral 
environment  is  very  complex  and  noisy,  traditional  ASW  tactics  and  systems  optimized 
for  the  open  ocean  do  not  work  effectively  in  littoral  waters.  High  noise  and  poor  sound 
propagation  in  the  littoral  waters  negatively  affect  the  effectiveness  of  the  underwater 
acoustic  sensors  that  are  developed  for  open-ocean  ASW  [16].  While  considering  the 
special  conditions  in  the  littoral  waters,  there  are  requirements  for  complementary 
capabilities.  A  new  technology  insertion  is  a  desirable  approach  to  achieve  and  improve 
current  and  near-term  ASW  capabilities  [16]. 

While  the  aim  of  littoral  ASW  operations  is  to  detect,  classify,  localize,  and 
neutralize  adversary  submarines,  there  will  be  a  need  to  employ  more  capable  ASW 
platforms,  proficient  operators,  and  reliable  sensor  systems  [14].  Modern  navies  employ  a 
variety  of  platforms,  such  as  surface  ships,  maritime  patrol  aircraft,  and  helicopters  for 
littoral  ASW  operations  and  coordinate  these  efforts  at  sea  to  complement  ASW 
capabilities  [16]. 
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2. 


ASW  Process 


Since  the  purpose  of  ASW  is  to  eliminate  the  submarine  threat,  the  ASW  process 
consists  of  several  phases.  In  general,  this  process  can  be  simplified  into  five  consecutive 
phases:  detection,  classification,  localization,  tracking,  and  kill  [18].  In  a  typical  scenario, 
ASW  assets  are  used  to  detect  and  classify  a  submarine  target,  hold  the  contact,  and  carry 
out  an  accurate  attack  (i.e.,  throw  weapons  or  depth  charges),  and,  if  necessary,  regain 
contact  and  re-attack  [19].  In  this  research,  the  effectiveness  of  an  ASW  screen  formation 
is  measured  by  the  proportion  of  successful  classifications.  Therefore,  we  touch  only  on 
the  detection  and  classification  phases.  These  initial  phases  must  be  successful  before  one 
can  localize,  track,  and  attack  a  submarine. 

Although  successful  ASW  requires  all  of  these  phases,  the  crucial  and  challenging 
phases  are  the  initial  detection  and  then  classification  of  a  submerged  submarine  hiding  in 
the  water.  Once  a  submarine  is  classified,  the  HVU  may  move  to  avoid  its  weapon  range. 
So,  the  success  of  an  ASW  operation  is  not  only  measured  by  the  destruction  of  the 
enemy’s  submarines  [20].  Indeed,  protecting  the  HVU  is  the  primary  ASW  objective. 

a.  Detection 

Detection  means  the  observation  of  an  underwater  contact,  which  may  be  a 
submarine  [18].  There  are  several  sensors  designed  to  detect  a  submarine.  We  divide 
these  sensors  into  two  basic  categories:  acoustic  sensors  and  non-acoustic  sensors.  While 
acoustic  sensors  pick  up  underwater  acoustic  signals  and  transfer  them  into  sound,  non¬ 
acoustic  sensors  use  various  techniques.  Acoustic  and  non-acoustic  sensors  include  active 
and  passive  sonars,  radar,  magnetic  anomaly  detection  (MAD),  electronic  support 
measure  (ESM)  devices,  and  sonobuoys  deployed  from  maritime  patrol  aircraft  (MPA). 
Visual  sighting  can  also  be  a  way  of  detecting  a  submarine. 

b.  Classification 

For  any  sonar  contact,  the  first  requirement  is  to  come  to  a  judgement  about  the 
contact.  This  judgment  is  called  classification  [18].  Classification  can  be  a  complicated 
phase  of  the  ASW  process,  but  it  is  very  important  to  categorize  whether  a  contact  is 
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related  to  a  submarine  or  not.  Contacts  are  classified  as  submarine,  non-submarine,  or 
doubtful.  Non-submarine  contacts  include  underwater  objects  such  as  sunken  ships,  sea 
creatures,  downed  aircraft,  or  lost  cargo.  If  these  underwater  objects  are  incorrectly 
classified  as  submarines,  it  causes  a  waste  of  time  and  effort  [18].  If  a  submarine  is 
wrongly  classified  as  non-submarine,  the  misclassification  could  threaten  and  damage 
HVUs  or  ASW  forces. 

In  tactical  situations,  a  diesel-electric  submarine  operates  underwater.  So,  it  is 
important  to  be  able  to  detect  it  there.  Since  overall  sonar  performance  is  degraded  in  the 
littorals,  this  platfonn  gains  extra  stealth  [21].  In  practice,  passive  sonar  is  not  effective  in 
noisy  littoral  environments  against  diesel-electric  submarines.  Active  sonar  is  the  best 
available  means  to  detect  and  classify  this  silent  threat  before  it  can  launch  a  torpedo. 

3.  ASW  Platforms 

This  section  discusses  types  of  ASW  platforms  as  well  as  the  combination  of  their 
properties  and  employment  methods.  A  variety  of  platforms,  including  surface  warships, 
rotary-wing  and  fixed-wing  aircraft,  submarines,  and  unmanned  vehicles  are  used  to 
localize  and  eliminate  enemy  submarines. 

There  are  some  common  capabilities  that  affect  the  success  of  ASW.  The  range  or 
reach  of  units  is  an  important  factor  in  ASW  as  well  as  in  ASuW  and  AAW.  Other 
important  factors  are  the  speed  and  endurance  of  the  units  [14].  When  an  ASW 
commander  makes  his  operations  plans,  he  considers  these  factors  and  knows  exactly  the 
strengths  and  the  weaknesses  of  the  various  ASW  platforms. 

Depending  on  the  given  task,  specific  platforms  will  be  assigned  to  form  the  ASW 
task  force.  Most  often,  two  or  more  escort  ships  and  their  organic  helicopters  are 
expected  to  accompany  the  HVU  if  threatened  by  a  diesel-electric  submarine, 

a.  Surface  Ships 

Surface  warships  have  many  warfare  capabilities  other  than  ASW.  The  most 
important  function  of  surface  warships  is  their  command,  control,  and  communication 
capabilities.  Because  the  payload  is  proportional  to  the  size  of  the  platform,  surface 
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warships  can  carry  a  large  number  and  variety  of  sensors  and  weapons — including  other 
ASW  platforms  such  as  helicopters  and  USVs. 

In  the  littoral  environment,  most  surface  warships  use  their  hull-mounted  active 
sonars.  Although  surface  warships  such  as  destroyers,  frigates,  and  littoral  combat  ships 
(LCSs)  have  a  speed  advantage  against  diesel-electric  submarines,  they  cannot  use  this 
advantage  effectively  in  ASW  as  speed  degrades  overall  sonar  performance  [14]. 

b.  ASW  Helicopters 

Helicopters  are  widely  used  in  ASW  operations  to  detect  and  eliminate  diesel- 
electric  submarines  hiding  under  temperature  inversions  in  the  water.  Helicopters  can  be 
deployed  from  surface  warships  and  extend  the  ships’  ASW  capability.  An  ASW 
helicopter  can  operate  without  detection  because  its  movements  cannot  be  seen  by  the 
submarine.  It  can  hover  above  the  surface,  lower  its  dipping  sonar  (variable  depth  sonar), 
and  operate  the  sonar  at  a  wide  variety  of  depths  (see  Figure  4).  In  this  manner  they  cover 
a  considerable  area  in  a  short  time,  providing  ASW  helicopters  a  great  advantage.  This 
advantage  is  generally  considered  as  a  characteristic  unique  to  helicopters  in  ASW.  These 
factors  provide  a  significant  capability  for  ASW  helicopters  as  a  screening  unit  ahead  of  a 
HVU  or  naval  convoy  [18]. 


Figure  4.  Aerial  view  of  an  SH-60F  Seahawk  helicopter  lowering  a  dipping 
sonar  into  the  Pacific  Ocean  (image  from  Wikimedia  Commons 
http  ://commons  .wikimedia,  org). 
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4. 


The  Acoustic  Environment 


The  underwater  environment  is  different  from  the  surface  environment.  Sound 
travels  unevenly  through  water  because  water  is  not  homogenous.  Both  passive  and 
active  sonar  perfonnance  are  significantly  affected  by  the  underwater  environment.  The 
measurements  of  temperature,  pressure,  and  salinity  all  change  at  different  layers  of  the 
water.  The  velocity  and  direction  of  sound  depends  on  all  of  these  factors.  The  changing 
acoustic  conditions  as  a  function  of  depth  create  a  considerable  bending  effect  on  sound 
waves  [18]. 

Temperature  is  the  most  significant  variable  that  affects  the  propagation  of  sound 
through  water.  Typically,  there  are  three  layers  at  sea  based  on  temperature:  mixed  layer 
(surface  layer),  thermocline,  and  deep  water.  The  mixed  layer  is  the  first  layer,  where  the 
temperature  is  almost  constant  with  depth.  The  second  layer  is  the  thermocline,  where 
temperature  changes  more  rapidly  with  depth.  The  last  one  is  the  deep  layer,  where 
temperature  decreases  very  slowly  with  depth  [18].  The  thermocline  layer  is  the  one  that 
we  are  interested  in.  When  sound  travels  into  the  thermocline,  it  tends  to  bend  and  creates 
shadow  zones  above  and  below  the  angle  of  the  sound  (see  Figure  5).  In  practice, 
submarines  know  where  the  thermocline  is  located  and  use  this  knowledge  to  hide  from 
surface  ships.  Submarines  pass  across  the  thermocline  layer  into  and  out  of  the  mixed 
layer  periodically  to  listen  for  targets.  This  factor  gives  diesel-electric  submarines  a  great 
concealment  capability. 
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Figure  5.  Thermocline  layer  effect  (image  from  http://weather.kopn.org). 


B.  UNMANNED  SURFACE  VEHICLES 

Unmanned  vehicles  have  inspired  great  interest  and  contributed  considerably  to 
military  operations  over  the  past  two  decades.  This  trend  is  likely  to  continue  into  the 
near  future.  Employing  unmanned  systems  in  military  operations  will  enhance  warfare 
capabilities  [22].  In  recent  years  unmanned  aerial  vehicles  (UAVs)  and  unmanned 
underwater  vehicles  (UUVs)  have  benefited  from  significant  research  and  development 
efforts.  USVs  have  received  relatively  less  focus  than  the  other  types  of  unmanned 
vehicles. 

1.  Overview 

According  to  the  U.S.  Navy’s  littoral  anti-submarine  warfare  concept,  “the 

accelerating  rate  of  technological  innovation  gives  increasing  advantages  to  the  navies 

that  most  quickly  introduce  appropriate  new  technologies  into  their  fleets”  [16]. 

According  to  a  report  of  the  Naval  War  College  Global  War  Game  in  2001,  “USVs  were 

key  contributors  in  establishing  situational  awareness  in  the  littorals  and  have  shown  the 

potential  to  provide  critical  access  to  high  risk  areas”  [23].  In  the  case  of  possible 
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conflicts  against  stealthier  enemies,  especially  in  littoral  waters,  putting  manned 
platforms  at  risk  is  no  longer  a  reasonable  course  of  action.  US  Vs  are  expected  to  be  a 
critical  complementary  element  of  modern  navies  in  the  future. 

USVs  have  some  significant  characteristics  that  can  complement  and  enhance 
current  warfare  capabilities:  reliability,  maneuverability,  long  endurance,  and  high 
payload  capacity.  These  primary  features  nominate  the  USV  as  a  complementary  element 
in  multiple  missions  [24],  Today,  modern  navies  are  looking  for  ways  to  use  these  risk- 
reducing  platforms  in  naval  missions,  especially  in  littoral  waters. 

In  2007,  the  U.S.  Navy  published  “The  Navy  Unmanned  Surface  Vehicle  (USV) 
Master  Plan”  [1].  This  master  plan  examines  the  capabilities,  classes,  and  potential  naval 
missions  for  USVs.  Seven  high-priority  USV  missions  are  identified  in  the  master  plan. 
These  missions,  in  priority  order,  are  [1] 

•  Mine  Countermeasures 

•  Anti-Submarine  Warfare 

•  Maritime  Security 

•  Surface  Warfare 

•  Special  Operations  Forces  Support 

•  Electronic  Warfare 

•  Maritime  Interdiction  Operations  Support 

According  to  open  online  sources,  the  U.S.  Navy  currently  has  four  classes  of 
USVs.  These  are  Fleet  Class  I,  Semi-Submersible  Snorkeling  Vessel,  Harbor  Class,  and 
Small  Class  [25].  Their  primary  missions  are  antisubmarine,  mine  countermeasures,  and 
surface  warfare  missions  for  the  littorals. 

2.  Development  of  the  Anti-Submarine  Warfare  Unmanned  Surface 

Vehicle 

In  recent  years,  advances  in  defense  technologies  have  offered  a  variety  of 
payloads  and  systems  for  USV  applications.  Potential  payloads  for  USV  systems  include 
towed  array  sonars,  dipping  sonars,  and  acoustic  sensors.  A  compact  dipping  sonar 
system  is  now  optimized  for  the  USV.  Therefore,  a  USV  can  take  advantage  of  the  same 
sensor  capability  as  ASW  helicopters. 
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General  Dynamics  Robotic  Systems  (GDRS)  developed  an  1 1  meter  “Fleet”  class 
Anti-Submarine  Warfare  Unmanned  Surface  Vehicle  (ASW  USV)  for  use  on  the  LCS 
and  delivered  the  first  one  to  the  U.S.  Navy  in  2008.  The  ASW  USV  is  autonomous  and 
capable  of  operating  in  an  extended-duration  with  a  high-payload  capacity.  It  has  high 
speed  capability  (35+knots),  thus  it  can  expand  the  reach  of  surface  warships. 
Characteristics  of  this  ASW  USV  are  shown  in  Table  1. 

Table  1.  Principal  characteristics  of  anti-submarine  warfare  unmanned 

surface  vehicle  (ASW  USV). 


Characteristic 

Characteristic 

Length 

40  ft 

Payload 

5000  lb 

Beam 

11.2ft 

Max  Speed 

35+  kt 

Max  Weight 

21,120  lb 

Endurance 

24+ hr 

3.  USV  Employment  for  Antisubmarine  Warfare 

Today’s  ASW  techniques  are  effective  in  most  cases,  but  employing  USVs  is 
likely  to  increase  the  effectiveness  of  ASW.  Employing  USVs  in  littoral  ASW  operations 
has  potential  to  enhance  the  efforts  of  existing  ASW  assets.  Effective  employment  and 
the  correct  tactical  use  of  USVs  offers  a  great  force  multiplier. 

U.S.  Navy  USV  Master  Plan  (2007)  defines  littoral  ASW  missions  in  three  major 
categories  (see  Figure  6):  “Hold  at  Risk,”  “Maritime  Shield,”  and  “Protected  Passage” 
[!]• 
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Figure  6.  Littoral  ASW  missions  in  three  major  categories. 


•  In  a  Hold  at  Risk  scenario,  USVs  monitor  for  submarines  in  the  entrance 
of  ports  or  chokepoints,  but  they  are  not  the  ideal  candidate  for  this 
category  due  to  their  limited  stealth. 

•  Maritime  Shield  missions  focus  on  clearing  a  Carrier  Strike  Group  (CSG) 
or  Amphibious  Ready  Group  (ARG)  operating  area  from  adversary 
submarines  and  keeping  that  area  secure. 

•  In  a  Protected  Passage  scenario,  USVs  clear  the  battlespace  of  enemy 
submarines  to  enable  secure  routes  for  an  Expeditionary  Strike  Group 
(ESG)  or  HVU. 

In  all  the  scenarios,  USVs  reduce  the  risk  to  manned  platforms  and  serve  as 
offboard  sensors,  thereby  extending  the  reach  of  warships.  A  warship  can  launch  a  USV 
and  serve  as  its  mother  ship. 

C.  AGENT-BASED  MODELING 

Agent-based  modeling  is  a  simulation  modeling  technique  that  has  been  used 
extensively  in  solving  real-world  problems,  including  military  applications  [26].  In  agent- 
based  modeling,  we  simulate  multiple  autonomous  decision-making  entities  called 
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agents.  Each  agent  makes  its  own  decisions  on  the  basis  of  a  set  of  user  defined  rules  and 
behaves  stochastically  in  its  local  environment  [27].  Agents  can  determine  their  behaviors 
with  their  predefined  personalities  and  be  aware  of  events  or  other  agents  by  using 
organic  or  inorganic  sensors. 

Agent-based  models  can  perform  non-linear  behavior  patterns,  capture 
organizational  dynamics,  and  provide  valuable  insights  about  real-world  systems  [26]. 
Military  applications  of  agent-based  simulations  are  widely  used  in  the  decision-making 
process.  Agent-based  simulations  can  capture  the  more  chaotic  and  intangible  aspects  of 
military  conflicts.  These  simulations  assist  decision-makers  in  testing  war  plans, 
reviewing  or  proposing  force  structures,  providing  detailed  information  on  today’s  high 
technology  products,  deciding  how  to  use  sensors  and  weapons,  and  exploring  potential 
changes  in  doctrine  or  tactics  [28]. 

There  are  many  simulation  tools  that  are  widely  used  for  agent-based  modeling. 
These  tools  include  general  computational  mathematics  systems  such  as  MATLAB  and 
Mathematica;  general  programming  languages  such  as  Python,  Java,  C++,  and  C;  and 
other  agent-based  modeling  platforms  such  as  NetLogo,  Swarm,  Repast,  AnyLogic, 
JANUS,  MAN  A,  and  Pythagoras  [29].  These  tools  are  used  in  different  fields  of  study 
and  real-world  applications.  Figure  7  displays  a  screen  shot  of  a  USV  scenario  in 
Pythagoras  [11]. 
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Figure  7.  A  screen  shot  of  a  USV  scenario  in  Pythagoras,  from  [1 1]. 


D.  MAP  AWARE  NON-UNIFORM  AUTOMATA  (MANA) 

The  simulation  tool  used  in  this  thesis  is  MANA,  which  is  developed  by  the 
Defence  Technology  Agency  in  New  Zealand.  MANA  has  been  widely  used  for  military 
and  academic  studies,  including  several  master’s  theses  at  the  Naval  Postgraduate  School. 
These  studies  include  maritime  protection  of  critical  infrastructure  assets  [30],  counter¬ 
piracy  escort  operations  in  the  Gulf  of  Aden  [31],  unmanned  aerial  vehicle  contributions 
for  expeditionary  operations  [32],  the  effectiveness  of  unmanned  aerial  vehicles  in 
helping  secure  a  border  [33],  and  the  operational  effectiveness  of  a  small  surface  combat 
ship  in  an  anti-surface  warfare  environment  [34]. 

MANA  is  designed  for  modeling  complex  adaptive  systems,  such  as  combat 
situations.  MANA  builds  time-stepped,  mission-level,  stochastic  simulations.  MANA 
contains  entities  representing  military  units  which  interact  with  their  environment  and  the 
other  entities  and  make  their  own  decisions.  Unlike  physics-based  models,  MANA  is 
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very  useful  to  simulate  and  analyze  the  effects  of  command  and  control,  situational 
awareness,  and  sensor  and  weapon  systems  [35].  Figure  8  shows  the  startup  screen  of 
MANA. 


MANA  V 


DTA 


,  ★ 
★  ★ 


DEFENCE  TECHNOLOGY  AGENCY 


Vector-Based  Movement 

Map  Aware  Non-uniform  Automata 


This  software  has  been  developed  by  Gregory  McIntosh.  David  Galliqan. 

Mark  Anderson  and  Michael  Lauren  for  the  New  Zealand  Army  and  Defence  Force. 

Terms  of  use: 

This  software  belongs  to  the  Defence  Technology  Agency  (DTA),  New  Zealand 
Defence  Force.  It  is  not  to  be  distributed  without  the  permission  of  DTA.  This 
product  is  not  to  be  offered  for  resale. 


View  end  user  license  agreement  (EULA1 


See  https://teams. nzdf.mil. nz/sites/mana  for  updates  and  support. 

(Requires  user  account.) 

I  understand  I  can  only  use  this  software  subject  to  the  conditions  stated  above. 
Version  5.01.09  I  Agree  Cancel 


pic:  J  Pee 


Figure  8.  The  startup  screen  for  MANA. 


MANA  modelers  have  the  ability  to  edit  battlefield  characteristics  and  create  a 
terrain  map  and  background  according  to  specific  scenarios.  Agents  behave 
independently  on  the  virtual  battlefield  based  on  their  personalities,  goals,  sensors, 
weapons,  and  terrain  type.  However,  they  will  not  respond  to  the  situations  in  the  same 
way  because  the  platfonn  is  stochastic  and  each  agent  uses  its  own  information  provided 
by  personal  sensors  or  communication  links  and  stored  in  organic/inorganic  SA  maps. 
Agents  can  also  have  completely  different  personalities  in  different  states  and  behave  in 
that  way  by  activating  trigger  states. 


23 


MANA  Version  4  User  Manual  defines  four  basic  parameters  that  affect  an  agent’s 
behavior  [36]: 

•  Personality  weightings  determine  an  agent’s  tendency  to  move  towards  or 
away  from  friendly,  neutral,  or  enemy  entities,  or  waypoints,  or  terrain. 

•  Move  constraints  are  meta-personalities  which  modify  an  agent’s  basic 
personality  weightings.  This  brings  an  agent  a  detailed  behavior  ability 
which  is  closer  to  the  reality. 

•  Intrinsic  capabilities  are  tangible  or  physical  characteristics  of  an  agent 
including  its  speed,  sensors,  weapons,  targeting  priorities,  and  fuel  level. 

•  Movement  algorithm  modifiers  affect  an  agent’s  speed  and  degree  of 
autonomy  when  moving. 

More  infonnation  can  be  found  in  MANA  Version  4  User  Manual  and  MANA-V 
(Map  Aware  Non-uniform  Automata-Vector)  Supplementary  Manual. 
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III.  MODEL  DEVELOPMENT 


“We  are  entering  an  era  in  which  unmanned  vehicles  of  all  kinds  will  take  on  greater 
importance  in  space,  on  land,  in  the  air  and  at  sea.  ” 

-  George  W.  Bush 

In  this  chapter,  a  brief  description  of  ASW  screen  formation  is  given,  as  well  as 
the  scenarios  used  for  this  thesis.  After  addressing  the  scenarios,  we  discuss  some  key 
modeling  assumptions  and  limitations.  Finally,  measures  of  effectiveness  and  model  stop 
conditions  are  explained. 

A.  ANTI-SUBMARINE  WARFARE  SCREEN  FORMATION 

The  purpose  of  defensive  ASW  operations  is  to  protect  a  convoy  of  ships  or 
HVUs  within  a  group  through  high-threat  areas.  Because  conventional  submarines  are 
serious  threats  to  HVUs  in  the  littoral  waters,  naval  operations  usually  focus  on  defensive 
ASW.  HVU  protection  requires  detailed  organization  and  a  carefully  set  formation.  A 
defensive  ASW  formation  (see  Figure  9)  is  generally  used  for  preventing  a  submarine 
from  reaching  a  position  around  an  HVU  from  which  it  could  launch  a  torpedo.  It  is 
necessary  to  use  acoustic  equipment  effectively  by  employing  highly  maneuverable 
surface  craft,  such  as  destroyers  and  frigates,  and  helicopters  at  an  effective  distance  from 
an  HVU  or  a  convoy  of  ships.  This  formation  is  generally  called  an  ASW  screen 
formation. 

The  screen  size  depends  on  the  availability  of  screening  vessels  in  the  ASW  task 
force.  If  a  large  force  is  available,  two  or  three  screens  may  be  employed  in  the 
formation.  One  or  two  screens  are  normally  used  for  small  forces.  There  are  three  classes 
of  ASW  screens  [37]: 

•  The  inner  ASW  screen  is  a  screen  in  which  surface  ships  position  around 
an  HVU  or  convoy  for  the  purpose  of  preventing  a  submarine  from 
reaching  the  torpedo  danger  zone. 
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•  The  intermediate  ASW  screen  is  a  second  screen  that  is  farther  away  from 
a  formation  of  ships,  has  the  potential  to  enhance  detection  and 
neutralization  capabilities. 

•  The  outer  ASW  screen  is  a  sound  screen  well  ahead  of  the  formation  of 
ships  and  HVU  for  the  purpose  of  detecting  the  approach  of  a  submarine 
and  alerting  the  assets  early. 


Example: 

"Operation 

Atlantic” 

Planning 
TG  200.1 

Screen  L  nits: 

2  Frigates 

1  Destroyer 

2  Helicopters 


Figure  9.  Possible  ASW  screen  formation,  from  [38]. 

The  inner  ASW  screen  is  the  most  important  one  among  these  three  classes.  The 
form  of  the  inner  ASW  screen  is  shaped  based  on  the  number  of  available  screening 
ships.  The  outer  screen  is  the  next  most  important  one,  and  ASW  helicopters  are 
generally  used  for  it.  If  screening  vessels  exceed  the  number  required  for  the  inner  and 
outer  ASW  screens,  the  intermediate  ASW  screen  may  be  employed. 

B.  SCENARIO  DESCRIPTIONS 

This  thesis  uses  the  combat  simulating  platform  called  MANA  to  model  the 
scenarios.  In  this  section,  the  battlefield  features  are  briefly  explained.  Then,  a  generic 
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ASW  scenario  is  created  to  increase  to  facilitate  exploring  USV  capabilities  and  tactics. 
Next,  we  describe  all  of  the  scenarios. 

1.  The  Battlefield 

The  battlefield  is  configured  as  40  nautical  miles  (nm)  wide  by  140  nm  long.  On 
this  battlefield,  our  area  of  interest  is  a  100  x  24  nm  box  in  which  MANA  positions  the 
enemy  submarine  randomly.  The  entire  battlefield  is  plain  terrain;  thus,  the  terrain  has  no 
effect  on  the  movements  of  the  agents.  In  this  model,  the  Cartesian  coordinate  system 
describes  all  positions  in  the  battlefield.  For  all  scenarios,  the  top  left-hand  comer  of  the 
battlespace  is  point  (0,  0),  and  the  bottom  right-hand  corner  is  point  (140,  40).  The 
battlefield  characteristics  are  shown  in  Figure  10. 

2.  Generic  Scenario 

A  Turkish  naval  task  force  (Blue)  has  been  tasked  to  move  from  an  area  of 
operation  to  another.  The  aim  of  this  task  force  is  to  transport  logistics  to  friendly  forces 
operating  at  sea.  This  task  force  consists  of  guided-missile  aviation  frigates  (FFGH), 
ASW  helicopters  (SH-70B),  and  unmanned  surface  vehicles  (USVs).  Their  main  goal  is 
to  protect  the  HVU,  a  mid-size  replenishment  oiler  (AOR).  Helicopters  and  USVs  are 
organic  to  the  frigates.  These  assets  can  be  deployed  from  the  frigates  and  generally 
operate  ahead  of  the  task  force. 

Intelligence  reports  warn  that  an  adversary  (Red)  diesel-electric  submarine 
threatens  the  SLOCs.  It  is  assumed  that  this  enemy  submarine  is  on  Blue’s  transit  routes, 
waiting  for  a  favorable  moment  to  engage  the  HVU  with  a  torpedo.  The  submarine 
selects  its  target  carefully;  it  almost  never  launches  a  torpedo  blindly  into  the  task  force. 
It  is  assumed  that  an  attack  on  ASW  assets  is  never  expected  because  the  diesel-electric 
submarine  desires  the  more  strategic  oiler  and  an  attack  on  an  escort  will  alert  its  primary 
target.  That  is,  the  submarine  will  not  put  its  life  at  risk  unless  it  can  fire  at  the  HVU. 

An  ASW  screen  is  fonned  to  detect  and  classify  a  submarine  when  a  task  force  is 
transiting  high- threat  areas.  The  deployment  tactic  plays  an  important  role  on  detection 
and  classification  of  the  submarine.  The  ASW  assets  try  to  detect  and  classify  the 
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submarine  before  it  penetrates  the  screen,  takes  a  planned  approach,  and  launches  a 
torpedo.  Once  the  submarine  is  classified,  normally,  the  ASW  task  force  attempts  to 
execute  the  localization,  tracking,  and  kill  phases.  However,  this  scenario  focuses  solely 
on  classifying  the  submarine  before  it  enters  the  torpedo  danger  zone  (TDZ)  around  the 
HVU.  Classifying  the  enemy  submarine  can  be  interpreted  as  reducing  the  risk  to  the 
HVU.  In  this  study,  the  ASW  process  after  the  classification  phase  is  not  simulated.  The 
overall  representation  of  the  generic  scenario  is  depicted  in  Figure  10. 


Figure  10.  The  battlefield  characteristics  and  the  overall  representation  of  the 

generic  scenario  (not  drawn  to  scale). 


The  baseline  and  advanced  scenarios  are  modeled  using  MANA.  The  scenarios 
were  built  to  explore  the  use  of  combinations  of  frigates,  helicopters,  and  USVs  to  protect 
an  HVU  from  a  single  enemy  submarine.  In  the  scenario  setup,  the  number  of  available 
frigates  ranges  from  two  to  three.  The  number  of  helicopters  and  USVs  are  dependent  on 
the  number  of  frigates,  which  serve  as  mother  ships  to  helicopters  and  USVs.  The  overall 
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scenario  description  is  shown  in  Table  2.  The  modeling  process  is  explained  in  simple 
language  in  the  following  sections. 


Table  2.  The  overall  scenario  description. 


Scenario 

ASW  Units 

Baseline  Scenario 

2FFGH 

2HELO 

- 

Scenario  Two 

2FFGH 

- 

2  USV 

Scenario  Three 

2FFGH 

2HELO 

2  USV 

Scenario  Four 

3  FFGH 

3  HELO 

- 

Scenario  Five 

3  FFGH 

- 

3  USV 

Scenario  Six 

3  FFGH 

3  HELO 

3  USV 

3.  Baseline  Scenario 

This  scenario  is  created  based  on  existing  ASW  screening  settings.  It  provides  us 
a  standardized  benchmark.  There  are  four  classes  of  agents  in  the  battlespace:  the  HVU, 
frigates,  ASW  helicopters,  and  the  enemy  submarine.  In  the  baseline  scenario,  the  HVU 
is  screened  by  two  frigates  and  two  organic  ASW  helicopters  because  it  does  not  have  an 
ASW  capability,  and  it  is  vulnerable  to  submarine  attacks.  The  frigates  are  equipped  with 
hull-mounted  sonars,  and  the  ASW  helicopters  are  equipped  with  dipping  sonars.  All 
equipped  vessels  are  using  their  sonars  in  active  mode.  The  submarine  listens  for  sound 
in  passive  mode. 

While  the  frigates  are  positioned  on  the  inner  ASW  screen,  the  helicopters  are 
positioned  on  the  outer  ASW  screen.  The  initial  locations  of  the  units  are  defined  using 
Cartesian  coordinates.  The  ASW  assets  are  initially  located  at  the  western  edge  of  the 
battlefield  outside  the  box.  The  coordinates  of  the  battlefield,  the  area  of  interest,  and  the 
initial  locations  of  the  units  are  depicted  in  Figure  11. 
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Figure  1 1 .  The  coordinates  of  the  battlefield,  the  area  of  interest,  and  the  initial 
locations  of  the  units  for  the  baseline  scenario  (not  drawn  to  scale). 


The  HVU  begins  at  the  point  (2,  20)  and  proceeds  as  a  moving  reference  point  at 
10  knots,  which  is  the  speed  of  advance  (SOA).  The  frigates  maintain  this  speed,  and 
their  movements  depend  on  the  HVU.  The  helicopters  are  initially  stationed  on  the 
frigates.  They  launch  from  their  mother  ships  and  move  to  the  first  dip  location  once  the 
simulation  starts.  Once  there,  they  hover  in  place  and  lower  their  sonar  transducers  into 
the  water. 

MANA  randomizes  the  initial  positions  of  the  agents  within  their  defined 
homeboxes.  Therefore,  we  can  expect  different  outcomes  each  time  the  model  is  run.  At 
initialization,  the  diesel-electric  submarine  is  positioned  randomly  by  MANA  in  the  area 
of  interest  and  thereafter  moves  randomly  at  3  knots.  When  it  becomes  aware  of  the  task 
force,  it  attempts  to  penetrate  the  ASW  screen  and  increases  its  speed  up  to  10  knots. 
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4. 


Scenario  Two 


In  this  scenario,  USVs  are  included  in  an  intermediate  screen  instead  of  the  outer 
screen  ASW  helicopters.  In  doing  so,  USVs  will  maintain  a  protective  ASW  barrier  in 
front  of  the  surface  group.  Referencing  the  coordinate  system  in  Figure  11,  the  starting 
locations  of  the  units  are  shown  as  follows: 

•  BlueHVU:  (2,20) 

•  BlueEscortl:  (7,26) 

•  BlueEscort2:  (7,  14) 

•  BlueUSVl:  (19,25) 

•  BlueUSV2:  (19,  15) 

USVs  carry  a  dipping  sonar  similar  to  the  one  used  by  the  helicopters.  The  USVs 
use  a  “Sprint  &  Drift”  tactic  ahead  of  the  mother  ship.  They  sprint  ahead  to  their  next  dip 
location,  and  once  there,  they  drift  on  the  water  and  lower  and  operate  their  dipping 
sonar. 


5.  Scenario  Three 

In  Scenario  Three,  we  update  the  baseline  scenario  again.  In  this  scenario,  all  of 
the  available  assets  are  deployed:  two  frigates,  two  ASW  helicopters,  and  two  USVs.  All 
of  the  agents  are  using  the  same  tactics  previously  discussed.  While  ASW  helicopters  are 
positioned  on  the  outer  ASW  screen,  USVs  are  positioned  on  the  intermediate  ASW 
screen.  The  HVU,  frigates,  and  USVs  are  located  at  the  same  starting  locations  as  in 
Scenario  Two.  Once  the  simulation  starts,  the  helicopters  are  deployed  ahead  of  the 
USVs. 


6.  Scenario  Four 

In  this  scenario,  the  HVU  is  screened  by  three  frigates  and  three  organic  ASW 
helicopters.  There  is  no  difference  between  this  scenario  and  the  baseline  scenario  in 
tenns  of  the  deployment  tactics  and  parameter  setup,  but  the  number  and  placement  of 
the  units  change.  The  initial  locations  of  the  units  are  shown  in  Figure  12. 
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Figure  12.  Scenario  Four:  The  initial  locations  of  the  units  (not  drawn  to  scale). 


7.  Scenario  Five 

In  Scenario  Five,  USVs  are  deployed  again  in  our  model  instead  of  ASW 
helicopters.  The  deployment  tactics  and  parameter  setup  are  the  same  as  before,  but  the 
initial  locations  and  the  sectors  relative  to  the  HVU  are  different.  Referencing  the 
coordinate  system  in  Figure  12,  the  starting  locations  of  the  units  are  shown  as  follows: 

•  BlueHVU:  (2,20) 

•  BlueEscortl:  (7,28) 

•  BlueEscort2:  (7,  12) 

•  BlueEscort3:  (8,  20) 

•  BlueUSVl:  (18,28) 

•  BlueUSV2:  (18,  12) 

•  BlueUSV3:  (18,20) 

8.  Scenario  Six 

In  Scenario  Six,  all  of  the  available  assets  are  deployed:  three  frigates,  three  ASW 
helicopters,  and  three  USVs.  All  of  these  agents  act  in  the  same  manner  as  in  previous 
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scenarios.  The  HVU,  frigates,  and  USVs  are  located  at  the  same  starting  locations  as  in 
Scenario  Five.  Once  the  simulation  starts,  the  helicopters  are  deployed  on  the  outer  ASW 
screen  ahead  of  the  USVs  in  an  intennediate  screen. 

C.  AGENT  DESCRIPTIONS 

MANA  agents  have  a  variety  of  tangible  characteristics,  such  as  agent  allegiance 
(friendly,  enemy,  or  neutral),  class  parameters,  threat  levels,  movement  speed,  and 
personal  concealment  rate. 

The  basic  assessment  of  an  agent’s  identity  is  that  of  allegiance.  Allegiance 
determines  the  side  of  an  agent.  We  define  the  allegiance  of  the  HVU,  frigates, 
helicopters,  and  USVs  as  friendly,  and  the  allegiance  of  the  diesel-electric  submarine  as 
enemy.  There  are  no  neutral  agents  in  our  scenarios.  We  also  added  stationary  dummy 
agents  that  simulate  random  dipping  locations  for  helicopters  and  USVs.  Their  allegiance 
is  defined  as  enemy  for  modeling  purposes  as  they  “attract”  the  helicopter  and  USV 
agents.  The  numeric  value  1  represents  blue  forces  and  2  represents  red  forces. 

Agent  class  parameters  and  agent  threat  levels  help  define  the  type  of  the  enemy. 
Agent  class  is  used  to  differentiate  the  target  types  for  weapon  engagement.  Because  we 
do  not  simulate  the  kill  phase  in  this  model,  a  dummy  weapon  model  is  used  for  stopping 
the  simulation  when  the  submarine  is  classified. 

The  threat  level  is  used  to  differentiate  the  target  types  on  the  situational 
awareness  maps  of  the  agents,  so  the  agent  can  react  to  that  information  according  to  user 
assigned  personality  weightings.  Table  3  shows  the  overall  tangible  characteristics  of  the 
agents. 
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Table  3.  The  tangible  characteristics  of  the  agents. 


Agent 

Description 

Allegiance 

Agent  Class 

Threat 

HVU 

High  Value  Unit 

1 

1 

3 

Escort  Ship 

Guided-Missile  Aviation 
Frigate  (FFGH) 

1 

2 

1 

Helicopter 

ASW  Helicopter 

1 

3 

1 

USV 

USV  equipped  with 
dipping  sonar 

1 

3 

1 

Submarine 

Conventional  Diesel- 
Electric  Submarine 

2 

4 

3 

Dipping  Agent 

Dummy  Enemy  Agent 

2 

94-99 

2 

1.  Friendly  Forces  Behaviors 

The  movement  behavior  of  an  ASW  unit  is  based  on  its  personality  weightings 
and  next  waypoint.  In  MANA,  the  personality  weightings  are  set  between  -100  and  100 
for  adjusting  the  directivity  of  the  agent.  For  more  details,  see  the  MANA  Version  4  User 
Manual  [36].  A  positive  weighting  value  attracts  an  agent  while  a  negative  value  repulses 
it.  The  modeler  can  play  with  the  weighting  values  to  obtain  the  desired  behavior. 

a.  HVU  and  Escort  Ships 

For  defining  the  movement  behavior  of  the  FIVU  and  the  escort  ships,  their 
personality  weightings  towards  the  next  waypoint  are  set  to  100.  Their  movement  toward 
the  waypoint  is  slightly  randomized  by  setting  the  random  patrol  bar  to  10.  This  adds  a 
small  amount  of  random  wiggle  to  their  movement,  as  with  real  platforms.  The  FIVU  uses 
just  the  Default  State  settings  because  it  does  not  change  its  behavior  during  the 
simulation.  Figure  13  shows  the  personality  weightings  and  trigger  states  of  the  FIVU. 
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Figure  13.  The  personality  weightings  and  trigger  states  of  the  HYU. 


The  escort  ships  have  four  states  created  to  simulate  helicopter  operations.  We  did 
this  by  applying  the  embussing  feature  in  MANA.  The  helicopters  are  carried  by  the 
escort  ships  until  a  release  trigger  point  is  reached.  In  the  beginning  of  the  simulation,  the 
escort  ships  change  their  state  from  Default  State  to  Run  Start.  In  this  state,  they  release 
their  child  squads,  that  is,  their  organic  helicopters.  After  the  duration  time  of  the  trigger 
state  passes,  the  escort  ships’  states  fall  back  to  the  Must  Embuss  state.  In  this  state,  the 
escort  ships  call  their  child  squads  back.  After  the  child  squads  arrive  at  the  escort  ships, 
they  station  there  during  the  Embussed  Children  state.  This  process  is  used  to  model  the 
endurance  of  the  helicopters.  Table  4  shows  the  trigger  states  of  the  escort  ships. 
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Table  4.  The  trigger  states  of  the  escort  ships. 


State 

Embussing  Behavior 

Next  State 

Default 

Nothing 

Run  Start 

Run  Start 

Release  Child  Squads 

Must  Embuss 

Must  Embuss 

Embuss  Children 

Embussed  Children 

Embussed  Children 

Nothing 

Run  Start 

b.  Helicopters  and  USVs 

The  helicopters  and  USVs  have  the  same  movement  pattern.  They  first  move  to 
the  nearest  dipping  location,  which  is  semi-randomized  in  the  area  of  interest.  Dummy 
enemy  agents  are  created  to  simulate  dipping  locations  and  randomized  in  their 
homeboxes.  The  homeboxes  are  set  to  a  reasonable  search  pattern.  The  threat  level  of 
these  agents  is  set  to  2,  and  they  attract  the  helicopters  and  USVs  in  their  Default  State. 
Once  a  helicopter  or  USV  finds  the  nearest  dummy  agent,  it  fires  at  this  agent,  and  then 
changes  its  state  to  Taken  Shot  (Sec).  In  this  state,  a  helicopter  hovers  over  the  water  and 
lowers  its  dipping  sonar  for  four  minutes.  Next,  its  state  falls  back  to  Spare  1  during 
which  it  enables  its  dipping  sonar.  Then,  it  recovers  its  dipping  sonar  for  four  minutes 
and  moves  forward  to  find  the  next  dummy  agent.  Table5  summarizes  this  process. 


Table  5.  The  trigger  states  of  the  helicopters  and  USVs. 


State 

Speed  (knots) 

Enable 

Sensor? 

(Yes/No) 

Duration 

(seconds) 

Next  State 

Helicopter 

USV 

Default 

Flight 

Speed 

Sprint 

Speed 

No 

Sprint  Time 

Taken  Shot 
(Sec) 

Taken  Shot  (Sec) 

0 

0 

No 

240 

Spare  1 

Spare  1 

0 

0 

Yes 

Dip  Time 

Spare  2 

Spare  2 

0 

0 

No 

240 

Default 
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2.  Enemy  Behaviors 

The  submarine  behavior  is  a  simple  process.  It  has  two  states:  Default  and  Enemy 
Contact.  In  the  default  state,  it  patrols  in  its  homebox  and  tries  to  detect  ASW  units.  A 
patrol  zone  is  created  by  using  the  random  patrol  feature  in  MANA.  This  allows  the 
submarine  to  travel  on  a  straight  path  on  random  routes  in  the  patrol  zone.  The  random 
patrol  settings  of  the  submarine  are  shown  in  Figure  14. 


Random  Patrol 

10  <  UJ  r 

Average  path  length: 

5000  g  metres 

V  Define  Patrol  Zone 

Left  =  20.0000  n.  miles 
Top  =  5.0000  n.  miles 
Width  =  100.0001  n.  miles 
Height  =  24.0000  n.  miles 

Figure  14.  The  random  patrol  settings  of  the  submarine. 

Once  the  submarine  detects  an  ASW  unit,  it  changes  its  state  to  Enemy  Contact.  It 
then  moves  through  the  center  of  the  formation  and  attempts  to  reach  the  TDZ.  The 
submarine  moves  forward  in  a  submerged  approach  region  and  attempts  to  remain 
undetected  to  reach  the  TDZ.  This  movement  is  set  with  several  changes  in  personality 
settings  of  the  agent.  The  personality  settings  of  the  submarine  in  Enemy  Contact  state 
are  shown  in  Figure  15. 
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Figure  15.  The  personality  settings  of  the  submarine  in  enemy  contact  state. 


3.  Sensor  Behaviors 

In  the  model,  sonar  is  the  only  detection  sensor  used  by  the  agents  since  the 
submarine  is  submerged.  The  escort  ships  use  their  hull-mounted  sonar  while  the 
helicopters  and  USVs  use  their  dipping  sonar  in  active  mode  to  detect  the  submarine.  The 
submarine  uses  its  hull-mounted  sonar  in  passive  mode  to  detect  the  ASW  assets.  While 
an  advanced  (probabilistic)  sensor  model  is  used  to  model  the  active  sonar  of  ASW  units, 
a  cookie-cutter  sensor  model  (see  Figure  16)  is  used  to  model  the  passive  sonar  of  the 
submarine. 
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Figure  16.  Cookie-cutter  sensor. 

Two  different  sensors  are  modeled  as  cookie-cutter  sensors  for  the  submarine’s 
passive  sonar.  One  sensor  type  is  modeled  for  counter-detection  of  ASW  units  that 
operate  active  sonar.  The  other  sensor  type  is  modeled  for  detecting  the  HVU,  which  does 
not  have  any  acoustic  sensors.  Because  counter-detection  of  active  sonars  can  be 
performed  at  greater  ranges  than  the  passive  sonar’s  detection  range  of  the  HVU,  the 
counter-detection  range  is  fixed  at  18,288  meters  (20,000  yards)  in  this  model  while  the 
detection  range  is  fixed  at  10,973  meters  (12,000  yards).  A  cookie-cutter  sensor  detects 
all  contacts  within  its  maximum  range.  Once  an  ASW  asset  enters  the  detection  range  of 
the  submarine,  MANA  records  the  detection  and  classification  with  the  probability  of  1.0. 
The  sensor  models  are  visualized  in  Figure  17. 
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1  ACTIVE  SONAR 

PASSIVE  SONAR 

FFGH 

usv 

Counter-detection 

HVU 

|J  FFGH 

USV 

Detection 

!  &  1 1 

|  Probabilistic  Sensor  Model 

Cookie  Cutter  Sensor  Model 

Figure  17.  Sensor  models. 


In  MANA,  sensor  models  for  the  advanced  sensor  type  are  defined  with  a 
detection  range-time  table  and  the  classification  range-probability  table  [36].  The 
detection  range-time  table  defines  the  average  time  between  detections  in  seconds  for  the 
specified  sensor  detection  range.  Figure  18  shows  an  escort  ship’s  sensor  setup  panel.  In 
this  panel,  the  detection  range  of  the  frigate  is  set  to  10,973  meters  (12,000  yards).  If  an 
underwater  contact  moves  in  the  detection  range  of  the  sensor,  the  frigate  will  detect  this 
contact  on  average  every  300  seconds — with  a  random  draw  each  time  step.  For  every 
detection  event,  the  submarine  has  a  chance  to  hide  in  the  water  based  on  its  personal 
concealment  rate.  Once  a  contact  has  been  detected,  the  ASW  unit  has  to  categorize 
whether  the  contact  is  related  to  a  submarine.  Detection  is  a  required  event  for  the 
classification  process  to  occur. 

The  classification  range-probability  table  determines  the  probability  of  classifying 
the  contact  for  the  specified  classification  range  once  the  detection  event  occurs.  In 
Figure  18,  the  classification  range  is  set  to  7,315  meters  (8,000  yards).  This  means  that  if 
the  submarine  is  in  this  range,  the  escort  ship  has  a  chance  to  classify  it.  If  the  submarine 
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is  in  the  detection  range,  but  out  of  the  classification  range,  the  detection  may  be 
successful,  but  the  classification  will  not  be. 


Figure  18.  Setup  panel  for  an  advanced  sensor  model. 


Detection  ranges  and  classification  range  intervals  for  all  the  agents  are 
summarized  in  Table  6. 
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Table  6.  Sensor  detection  ranges  and  classification  range  intervals. 


Sonar  Type 

Detection 

Range 

Classification  Range 
Interval 

Minimum 

Maximum 

Ship  hull-mounted  sonar 

12000  yards 

6000  yards 

10000  yards 

Dipping  sonar 

12000  yards 

4000  yards 

10000  yards 

Submarine 

hull-mounted 

sonar 

Counter-detection 

20000  yards 

20000  yards 

Detection 

15000  yards 

15000  yards 

D.  STOP  CONDITIONS 

Stop  conditions  were  introduced  to  the  model  to  reduce  runtime.  The  simulation 
stops  when  one  of  the  following  conditions  happens  (see  Figure  19): 

•  The  submarine  is  classified  by  one  of  the  ASW  units; 

•  The  submarine  reaches  the  TDZ  around  the  HVU;  or 

•  The  FIVU  reaches  its  final  waypoint. 


Figure  19.  Stop  conditions. 


42 


E.  SCENARIO  ASSUMPTIONS  AND  LIMITATIONS 

Scenario  assumptions  and  limitations  are  vital  to  a  successful  study.  It  is 
necessary  to  make  acceptable  assumptions  and  define  limitations  to  create  a  model 
realistic  enough  to  obtain  useful  insights. 

1.  Assumptions 

a.  Friendly  Forces 

(1)  USVs  are  launched  from  surface  warships  and  they  are  fully  autonomous. 

(2)  USVs  meet  autonomous  requirements,  such  as  station-keeping. 

(3)  Helicopters  and  USVs  use  a  Sprint  &  Drift  tactic.  They  sprint  ahead  to  the 
next  dip  location,  and  once  there,  they  drift  or  hover  on  the  water  and 
lower  and  operate  their  dipping  sonar. 

(4)  Dipping  points  are  semi-randomized  in  the  area  of  interest. 

(5)  Once  an  ASW  unit  detects  an  underwater  contact,  it  can  execute  the 
classification  process  itself. 

(6)  Each  unit  has  a  chance  of  classifying  the  submarine  for  every  detection 
event. 

b.  Enemy 

(1)  The  submarine  operates  submerged  during  the  simulation.  By  doing  this,  it 

minimizes  detection  by  the  ASW  forces. 

(2)  The  submarine’s  initial  position  is  selected  at  random  in  its  homebox. 

(3)  The  submarine  does  not  attack  frigates,  helicopters,  or  USVs.  Its  only 

target  is  the  HVU. 

2.  Limitations 

We  defined  the  limitations  of  MANA  when  building  the  model.  These  limitations 
must  be  considered  in  the  analysis  chapter.  The  first  and  most  important  one  is  that  it  is 
very  hard  to  implement  an  advanced  naval  formation,  such  as  an  ASW  screen  fonnation, 
in  MANA.  Another  limitation  is  that  the  level  of  classification  is  limited  to  a  binary 
response:  0  or  1 .  For  us,  0  represents  the  levels  non-submarine  (NONSUB)  and  doubtful 
(POSSUB  and  PROBSUB)  levels,  and  1  represents  the  certain  submarine  (CERTSUB) 
level.  Also,  it  is  difficult  to  simulate  the  underwater  environment.  The  changes  in  the 
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environmental  conditions  are  simulated  by  varying  detection  chances  and  classification 
probabilities.  Finally,  the  submarine’s  actual  depth  is  not  explicitly  simulated.  The 
submarine  can  hide  below  thennal  layers,  beneath  undersea  mountains,  or  on  the  sea 
floor.  The  submarine’s  concealment  rate  per  detection  event  accounts  for  the  submarine’s 
stealthiness. 
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IV.  MODEL  EXPLORATION 


A.  DESIGN  OF  EXPERIMENTS 

The  Design  of  Experiments  (DOE)  is  a  practical  approach  for  large-scale 
experiments  to  examine  design  factors  and  determine  the  relationship  between  design 
factors  and  output  responses.  In  experimental  tenninology,  design  factors  are  the  input 
variables,  and  output  responses  are  the  measures  of  effectiveness  or  performance  [39]. 

Although  cluster  computers  can  run  simulations  very  quickly,  it  is  an  impossible 
task  to  run  all  possible  design  points.  The  quality  of  the  results  can  be  determined  by  the 
model  runs.  An  efficient  design  is  needed  to  analyze  a  sufficient  breadth  of  possible 
outcomes.  Otherwise,  we  may  limit  the  insights  in  the  analysis. 

In  this  thesis,  a  nearly  orthogonal  Latin  hypercube  (NOLH)  spreadsheet 
developed  by  Susan  Sanchez  is  used  to  generate  the  design  points  [40].  The  advantage  of 
using  an  NOLH  design  is  that  it  has  good  space-filling  properties  and  meets  the 
orthogonality  criteria  necessary  for  good  statistical  properties  of  analysis  methods.  This 
design  can  provide  efficient  information  about  the  experiment.  A  well-designed  NOLH 
allows  the  analyst  to  efficiently  explore  more  factors  across  the  design  space  and  fit  a 
variety  of  diverse  models  to  multiple  different  response  variables.  A  scatterplot  matrix  in 
Figure  20  shows  the  space-filling  properties  of  our  NOLH  design. 

The  NOLH  design  spreadsheet  allows  us  to  create  an  efficient  design  and  saves 
time  and  effort.  Different  designs  are  available  on  this  spreadsheet  based  on  the  number 
of  design  factors.  A  design  with  more  design  points  is  a  favorable  thing,  but  not  required. 
We  choose  the  16-factor  design  to  build  our  experimental  design.  The  design  points  used 
in  this  study  are  provided  in  Appendix  A. 
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Figure  20.  Scatterplot  matrix  for  the  design  factors. 


B.  DESIGN  FACTORS 

In  an  ASW  scenario,  many  factors  may  affect  the  outcome.  In  total,  16  factors 
were  varied  in  the  simulation  for  each  scenario.  Design  factors  can  be  divided  into  two 
groups:  controllable  and  uncontrollable  factors.  These  factors  are  varied  over  a  range  in 
order  to  explore  their  effects  on  the  outcome.  They  are  explained  in  the  following 
sections.  Table  7  shows  the  list  of  factors  with  their  ranges,  units,  and  explanations. 
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Table  7.  Description  of  controllable  and  uncontrollable  factors. 


Factor 

Explanation 

Minimum 

Maximum 

Unit 

Controllable  Factors 

FleloSpeed 

Flelicopter  Speed  at  Default 

State 

80 

120 

knot 

USVSpeed 

USV  Speed  at  Default  State 

20 

35 

knot 

FleloAvgDetTime 

Flelicopter  Sonar  Average  Time 
Between  Detections  at  Default 
State 

180 

360 

second 

USVAvgDetTime 

USV  Sonar  Average  Time 
Between  Detections  at  Default 
State 

180 

360 

second 

FleloEndTime 

Flelicopter  Endurance  (Duration 
Time  of  Escort  Ships’  Run  Start 
State) 

5400 

9000 

second 

FleloRefTime 

Flelicopter  Refuel  Time 
(Duration  Time  of  Escort 

Ships’  Embussed  Children 

State) 

1200 

2400 

second 

FleloDipTime 

Flelicopter  Dipping  Time 
(Duration  Time  of  Flelicopters’ 
Spare  1  State) 

300 

600 

second 

USVDipTime 

USV  Dipping  Time  (Duration 
Time  of  USVs’  Sparel  State) 

300 

600 

second 

FrigateClassify  Range 

Frigate  Sonar  Classification 
Range 

5486 

9144 

meter 

FleloClassify  Range 

Flelicopter  Sonar  Classification 
Range 

3658 

9144 

meter 

US  VC  lassify  Range 

USV  Sonar  Classification 

Range 

3658 

9144 

meter 

FrigateClassify  Prob 

Frigate  Classification 

Probability 

0.2 

0.4 

- 

FleloClassify  Prob 

Flelicopter  Classification 
Probability 

0.4 

0.6 

- 

U  S  V  Classify  Prob 

USV  Sonar  Classification 
Probability 

0.4 

0.6 

- 

Uncontrollable  Factors 

SubAttackSpeed 

Submarine  Attack  Speed  at 
Enemy  Contact  State 

5 

10 

knot 

SubConcealment 

Submarine  Personal 

Concealment  per  Detection 

40 

60 

% 
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1. 


Controllable  Factors 


Controllable  factors  are  related  to  the  decisions  of  friendly  assets,  which  can  be 
decided  upon  in  advance  or  during  the  mission.  In  this  model,  they  are  all  related  to  the 
characteristics  of  the  frigate,  helicopter,  and  USV.  Controllable  factors  included  the 
movement  speed,  dipping  time,  endurance  time,  refuel  time,  average  time  between 
detections,  classification  range,  and  classification  probability. 

a.  Movement  Speed 

The  movement  speed  is  varied  in  the  experimental  design  to  determine  whether 
this  factor  has  an  effect  on  mission  success.  Because  the  most  considerable  strength  of 
the  helicopter  is  its  high  speed  capability,  it  can  execute  the  dipping  process  easily  over  a 
large  area.  However,  its  endurance  is  limited.  Once  the  helicopter  runs  out  of  fuel,  it 
moves  back  to  the  mother  ship,  refuels,  and  deploys  to  the  station  again.  Four  states  are 
defined  to  simulate  the  dipping  process.  In  “Default  State,”  a  helicopter  moves  to  the  next 
dipping  location  with  a  speed  of  80  to  120  knots.  In  a  trigger  state,  the  speed  of  the 
helicopter  is  set  to  zero  since  it  hovers  over  the  water. 

The  helicopters  are  significantly  faster  than  the  USVs.  This  factor  may  give  the 
helicopters  an  advantage  over  USVs  in  terms  of  mobility.  USVs  execute  the  dipping 
process  the  same  way  as  the  helicopter.  When  a  USV  is  in  “Default  State,”  it  moves  to 
the  next  dipping  location  with  a  speed  of  20  to  35  knots.  In  a  trigger  state,  it  drifts  on  the 
water  with  zero  speed. 

b.  Sensors 

The  main  focus  of  the  defensive  ASW  operations  is  the  ability  to  detect  and 
classify  the  enemy  submarine.  Since  using  active  sonar  is  the  primary  method  to  detect 
and  classify  the  diesel-electric  submarine,  the  frigate  uses  a  hull-mounted  sonar,  and  the 
helicopter  and  USV  use  dipping  sonars  in  active  mode.  The  sensor  parameters  are  mostly 
controllable  because  the  selection  of  the  sensor  type  and  capability  can  be  decided  during 
the  design  process.  But,  the  effectiveness  of  a  sonar  is  limited  by  environmental 
conditions.  In  our  model,  we  consider  the  sensor  parameters  only  partly  controllable. 
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The  helicopter  and  USV  use  the  same  sensor  type  in  the  model.  The  perfonnance 
of  the  sonar  for  each  platform  can  be  evaluated  by  varying  three  factors:  the  time  interval 
between  consecutive  detections,  classification  range,  and  classification  probability.  These 
factors  relate  to  the  helicopter  and  USV  varied  independently  over  the  same  range.  So, 
we  can  explore  three  different  cases:  a  better  sonar  perfonnance  for  the  USV,  a  better 
performance  for  the  helicopter,  and  the  same  perfonnance  for  both  platforms. 

We  define  the  time  interval  between  consecutive  detections  as  the  period  between 
the  event  initiation  and  completion.  This  is  a  simplification  of  simulating  detection 
chances.  The  other  states  are  locked  to  the  default  state,  so  varying  the  average  time 
between  detections  in  the  default  state  is  enough.  Thus,  it  is  desirable  to  have  the  mean 
detection  time  as  small  as  possible. 

The  classification  ranges  of  the  platforms  provide  a  reduced  danger  area.  If  the 
platforms  have  a  short  classification  range,  then  the  submarine  has  a  good  chance  of 
penetrating  the  ASW  screen.  In  ASW,  the  classification  range  depends  on  underwater 
conditions,  background  noise  in  the  ocean,  and  sonar  capability. 

The  other  sensor  parameter  is  the  classification  probability.  In  ASW,  this  factor 
may  depend  on  target  characteristics  or  the  training  of  the  operators.  For  USVs,  it  may 
depend  on  the  development  and  perfonnance  of  automatic  detection  and  classification 
systems  and  techniques. 

c.  Tactical  Employment  of  ASW  Assets 

Six  scenarios  were  built  to  explore  the  use  of  the  combinations  of  frigates, 
helicopters,  and  USVs.  The  name  of  the  scenarios  is  viewed  as  a  categorical  factor  for  a 
quick  comparison  of  the  scenarios.  We  expanded  this  factor  into  three  different 
categorical  factors  for  partition  tree  and  regression  analysis:  the  number  of  available 
frigates,  helicopter  presence,  and  USV  presence.  These  factors  relate  to  the  tactical 
employment  of  the  helicopters  and  USVs  as  well  as  the  design  of  the  ships.  The  factors 
related  to  the  scenarios  are  shown  in  Table  8. 
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Table  8.  The  factors  related  to  scenario  setup. 


Scenario 

The  Number 
of  Available 
Frigates 

Flelicopter 
Presence 
(l=Yes,  0=No) 

USV  Presence 
(l=Yes,  0=No) 

Baseline  Scenario 

2 

1 

0 

Scenario  Two 

2 

0 

1 

Scenario  Three 

2 

1 

1 

Scenario  Four 

3 

1 

0 

Scenario  Five 

3 

0 

1 

Scenario  Six 

3 

1 

1 

2.  Uncontrollable  Factors 

Uncontrollable  factors  are  related  to  the  enemy  and  uncertainty  in  the  combat 
environment.  There  are  two  factors  regarding  the  enemy  submarine:  attack  speed  and 
personal  concealment  per  detection  opportunity. 

a.  Speed 

The  submarine  patrols  at  3  knots  in  its  homebox.  When  it  detects  an  ASW  unit,  it 
changes  its  state  to  “Enemy  Contact  State”  and  tries  to  penetrate  the  ASW  screen  by 
increasing  its  speed.  The  speed  of  the  submarine  in  “Enemy  Contact  State”  is  a  factor  that 
ranges  from  5  to  10  knots. 

b.  Stealth 

The  submarine  is  designed  to  submerge  and  maneuver  quietly  to  avoid  detection. 
In  a  noisy  littoral  environment,  the  submarine  gains  extra  stealth.  The  submarine  can  find 
shadow  zones  to  hide  from  active  sonar  and  approach  an  enemy  without  being  detected. 
This  factor  was  simulated  in  MANA  using  the  personal  concealment  per  detection 
feature.  This  factor  represents  a  probability  of  stealth  per  detection  event.  The  stealth  of 
the  submarine  is  varied  between  40%  and  60%  in  the  experimental  design. 
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C.  DATA  ANALYSIS 

After  explaining  the  model  development,  experimental  design,  and  design  factors, 
we  now  focus  on  data  analysis.  In  this  section,  model  runs,  our  analysis  tools,  and 
measures  of  effectiveness  are  described  briefly.  This  is  followed  by  a  comparison  of  the 
scenarios,  regression  analysis,  and  partition  trees. 

1.  Model  Runs 

Using  the  NOLH  design  for  16-factors,  65  design  points  were  created  for  each 
scenario.  Each  design  point  was  run  1,000  times,  resulting  in  65,000  simulated  ASW 
missions  for  each  scenario.  The  time  step  in  this  model  is  fixed  to  one  second  because  it 
was  observed  that  large  time  steps  led  to  unusual  behaviors.  Since  there  are  six  different 
scenarios  being  evaluated,  a  total  of  390,000  runs  were  executed.  On  average,  each  model 
run  takes  approximately  one  minute  of  computer  runtime  on  a  personal  computer.  As 
expected,  the  more  agents  that  are  included  in  the  model,  the  longer  the  runtime  it  takes. 
For  example,  the  scenarios  with  three  frigates  take  more  time  to  run  than  the  scenarios 
with  two  frigates.  This  is  because  each  additional  agent  and  its  interactions  with  the 
others  will  require  considerably  larger  computational  effort. 

On  a  personal  computer  it  would  take  approximately  250-300  days  to  complete 
this  experiment  and  get  the  data.  Fortunately,  the  Naval  Postgraduate  School’s 
Simulation  Experiments  &  Efficient  Design  (SEED)  Center  offers  a  great  computational 
resource  for  thesis  students.  The  SEED  Center  can  use  over  a  hundred  processors  in 
parallel  to  make  MANA  runs.  With  this  advantage,  all  of  the  runs  were  completed  and 
the  data  was  synthesized  into  a  single  comma-separated  (CSV)  file  in  just  a  few  days. 

2.  Analysis  Tool 

The  analysis  tool  used  in  this  study  is  JMP,  a  statistical  analysis  tool  developed  by 
the  JMP  business  unit  of  SAS  Institute.  It  is  very  useful  to  the  analyst  for  investigating 
and  exploring  the  data.  This  software  is  used  to  interpret  the  data  by  performing  analyses 
and  creating  graphs,  data  tables,  charts,  and  reports.  JMP  automatically  displays 
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statistical  text  as  well  as  graphs  and  charts;  this  makes  it  a  user-friendly  analysis  tool.  The 
edition  utilized  in  this  study  is  JMP  Pro  12. 

3.  Measure  of  Effectiveness 

In  this  study,  mission  success  is  considered  classifying  the  submarine  before  it 
enters  the  TDZ  around  the  HVU.  Two  measures  can  be  defined  to  represent  this  goal:  the 
proportion  of  successful  classification  and  the  time  to  classify  the  submarine.  The  first 
MOE  is  the  success  rate,  which  represents  the  proportion  of  classification.  The  result  of 
each  run  is  a  binary  output:  1  or  0.  “1”  means  that  the  submarine  is  classified  and  “0” 
means  that  the  underwater  contact  isn’t  detected  or  isn’t  classified  as  a  submarine  before 
the  submarine  enters  the  TDZ.  The  average  of  the  binary  data  gives  us  the  overall 
proportion  of  Is  in  the  output  for  each  design  point.  If  an  ASW  asset  classifies  the 
submarine,  we  assume  this  reduces  the  risk  for  the  HVU.  We  also  assess  the  effectiveness 
of  each  scenario  by  quantifying  the  time  to  classify  the  submarine.  This  measure  is 
defined  as  our  second  MOE.  That  is,  the  earlier  a  submarine  is  detected,  the  better  it  is  for 
the  defenders.  In  some  cases,  early  detection  and  classification  is  critical  for  decision¬ 
makers  since  it  plays  an  important  role  in  keeping  the  HVU  out  of  harm’s  way. 

4.  A  Quick  Comparison  of  the  Scenarios 

Two  MOEs  are  used  for  evaluating  the  effectiveness  of  deployment  tactics.  The 
first  MOE  is  the  proportion  of  mission  success,  which  represents  the  overall  probability 
of  classification  before  the  submarine  reaches  the  TDZ.  The  second  MOE  is  the  time  to 
classify  the  submarine. 

A  single  CSV  file  with  390,000  rows  of  raw  data  is  imported  into  the  analysis  tool 
JMP.  First,  we  create  a  summary  data  table  with  all  the  raw  data  by  averaging  the  MOE1 
(mission  success)  column  for  each  scenario.  For  the  Baseline  Scenario,  the  mean  of 
success  in  the  overall  replications  is  around  0.383.  The  success  rate  in  the  other  scenarios 
differs  due  to  the  number  and  variety  of  the  platforms.  Scenario  Six  gives  the  highest 
success  rate,  since  more  platforms  are  employed  than  in  the  other  scenarios.  Table  9 
shows  the  success  rate  based  on  different  scenarios. 
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Table  9.  The  proportion  of  successful  classification  in  the  overall 

replications. 


Scenario 

Name 

Number  of 
Replication 

Number  of 
Success 

Success 

Rate 

Baseline  Scenario 

65000 

24953 

0.383892308 

Scenario  Two 

65000 

24578 

0.378123077 

Scenario  Three 

65000 

30752 

0.473107692 

Scenario  Four 

65000 

29108 

0.447815385 

Scenario  Five 

65000 

28664 

0.440984615 

Scenario  Six 

65000 

34947 

0.537646154 

The  scenarios  are  grouped  to  create  different  datasets.  Because  the  factors  and 
their  ranges  are  identical  for  all  scenarios,  we  can  directly  compare  them.  From  this 
comparison,  we  can  determine  how  a  change  in  the  configuration  of  ASW  assets  affects 
the  MOE.  Side-by-side  box  plots  are  particularly  useful  when  comparing  different 
datasets.  They  provide  us  a  quick  comparison  of  the  scenarios.  This  comparison  can  help 
a  tactical  commanders  choose  an  appropriate  configuration  of  ASW  assets.  For  each 
scenario,  a  box  is  created  extending  from  the  25th  percentile  to  the  75th  percentile.  The 
50th  percentile  is  the  median,  which  is  drawn  inside  the  box.  Whiskers  are  the  lines  that 
limit  a  subset  of  the  data,  outside  the  box. 

We  created  two  different  side-by-side  box  plots:  the  proportion  of  successful 
classification  versus  scenario  and  time  steps  to  classification  versus  scenario.  Figure  21 
displays  a  comparison  of  the  average  mission  success  versus  scenario.  The  box  plot  for 
Scenario  Six  gives  higher  results  on  the  average  mission  success  scale  than  the  others. 
While  the  Baseline  Scenario  and  Scenario  Two  look  similar  to  each  other,  Scenario  Four 
and  Scenario  Five  also  look  similar. 
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Mean(Success)  vs.  Scenario 
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Figure  21.  Comparative  boxplots:  Mean(success)  vs.  scenario. 

We  basically  see  that  while  assuming  the  helicopter  and  USV  have  the  same 
sensor  type  and  capability,  the  classification  effectiveness  of  the  ASW  screen  will  be 
approximately  the  same.  So,  we  turn  to  the  second  MOE,  time  to  classify,  to  see  if  there 
is  a  difference.  We  take  a  subset  of  data  for  every  level  of  the  Mission  Success  column. 
This  created  two  different  data  tables:  Mission  Success=l  and  Mission  Success=0. 
Between  them,  the  Mission  Success=l  data  table  is  the  one  that  we  will  use  to  quantify 
the  second  MOE.  The  Steps  column  gives  the  number  of  time  steps  in  a  scenario  until  the 
submarine  is  classified.  The  distributions  of  the  “Steps”  column  by  scenarios  are 
provided  in  Appendix  B.  Figure  22  displays  a  comparison  of  the  average  steps  versus 
scenario.  When  we  look  at  the  side-by-side  box  plot,  firstly,  we  realize  that  the  box  plots 
of  Scenario  Three  and  Scenario  Six  resemble  each  other  and  that  the  average  time  steps 
are  significantly  less  than  in  the  other  scenarios.  Employing  the  helicopters  and  USVs 
together  in  an  ASW  screen  formation  will  give  us  an  early  detection  and  classification 
capability.  The  early  detection  and  classification  of  the  submarine  is  a  crucial  factor  in 
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ASW  because  they  enable  the  task  force  commander  to  easily  keep  the  HVU  outside  of 
the  danger  zone  of  the  enemy  submarine. 


Mean(Steps)  vs.  Scenario 
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Figure  22.  Comparative  boxplots:  Mean(steps)  vs.  scenario. 


5.  One-way  Analysis  of  the  Means  by  Scenarios 

A  /-test  is  used  to  examine  the  difference  between  two  means  and  assumes  that 
the  samples  are  randomly  drawn  from  normal  populations;  though  the  test  is  robust  to 
nonnormality).  In  this  study,  the  six  scenarios  are  independent;  therefore,  another  way  of 
comparing  the  scenario  means  is  by  using  a  /-test.  In  this  test,  we  use  a  significance  level 
of  a  <  0.05. 

a.  The  Proportion  of  Successful  Classification 

The  scenarios  were  built  to  explore  how  different  combinations  of  frigates, 
helicopters,  and  USVs  contribute  to  the  detection  and  classification  of  the  submarine.  The 
proportion  of  classification  is  the  first  measure  for  the  comparison  of  the  scenarios.  We 
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use  JMP  to  perform  multiple  pairwise  comparisons  of  group  means.  Figure  23  shows  the 
visual  comparison  of  the  scenario  means  in  terms  of  the  proportion  of  classification. 
Interpreting  the  comparison  circles  is  a  basic  way  to  compare  group  means.  If  the 
comparison  circles  for  different  scenarios  do  not  intersect  or  intersect  slightly,  the  means 
of  the  scenarios  are  statistically  significantly  different.  If  the  comparison  circles  for 
different  scenarios  intersect  or  intersect  by  an  angle  of  higher  than  90  degrees,  the  means 
of  the  scenarios  are  not  significantly  different.  From  Figure  23,  we  can  interpret  that  the 
Baseline  Scenario  and  Scenario  Two  are  not  significantly  different.  Scenario  Six  (with 
three  of  all  the  assets)  is  the  only  one  that  is  significantly  different  than  all  the  other 
scenarios. 
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Figure  23.  The  visual  comparison  of  the  scenario  means  in  terms  of  the 

proportion  of  classification. 


From  the  detailed  results,  as  shown  in  Figure  24,  we  can  see  that  there  are  four 
comparisons  among  all  pairwise  comparisons  where  no  statistically  significant  difference 
is  found: 


•  Baseline  Scenario  where  two  frigates  and  two  helicopters  are  employed 
and  Scenario  Two  where  two  frigates  and  two  US  Vs  are  employed. 
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•  Scenario  Four  where  three  frigates  and  three  helicopters  are  employed 
and  Scenario  Five  where  three  frigates  and  three  USVs  are  employed. 

•  Scenario  Three  where  two  frigates,  two  helicopters,  and  two  USVs  are 
employed  and  Scenario  Four  where  three  frigates  and  three  helicopters 
are  employed. 

•  Scenario  Three  where  two  frigates,  two  helicopters,  and  two  USVs  are 
employed  and  Scenario  Five  where  three  frigates  and  three  USVs  are 
employed. 

The  other  pairwise  comparisons  show  that  there  is  a  statistically  significant 
difference.  A  detailed  report  that  compares  each  pair  is  provided  in  Appendix  C. 


A  Means  Comparisons 

A  »  Comparisons  for  each  pair  using  Student's  t 
A  Confidence  Quantile 
t  Alpha 

i.966i6  Qx) — ►  Significance  Level 

«d  LSD  Threshold  Matrix 

Abs(Dif)-LSD 


Scenario_6  Scenario_3  Scenario_4  Scenario_5  Basehne_Scenario  Scenario.2 


Scenario_6 

•0.03473 

0.02981 

0.05510 

0.06193 

0.11902 

0.12479 

Scenario.3 

0.02981 

-0.03473 

-0.00944 

-0.00261 

0.05449 

0.06025 

Scenario.4 

0.05510 

-0.00944 

-0.03473 

-0.02790 

0.02919 

0.03496 

Scenano_5 

0.06193 

-0.00261 

-0.02790 

-0.03473 

0.02236 

0.02813 

Baseline_Scenario 

0.11902 

0.05449 

0.02919 

0.02236 

-0.03473 

-0.02896 

Scenario.2 

0.12479 

0.06025 

0.03496 

0.02813 

;  Q2M 

-0.03473 

In  LSD  Threshold 
Matrix,  positive 
values  indicate  that 
there  is  statistically 
significance 
between  pairs 


Positive  values  show  pairs  of  means  that 


Connecting  Letters  Report 

Level 

$cenario 6  A 

0.53764615 

Scenario 3  B 

0.47U0769 

ScenariojJ  B 

0.44  781538 

0.44008462 

Baseline.Scenario  C 

0.38389231 

Scenario  2 - C- 

0.37812308 

’  significantly  different. 

The  group  means  that  are 
not  sharing  the  same 
letter  are  considered  as 
significantly  different 


Levels  not  connected  by  same  letter  are  significantly  different 


Ordered  Differences  Report 


Level 

-  Level 

Difference 

Std  Err  Dif 

Lower  CL 

Upper  CL 

p  Value 

Scenario_6 

Scenario_2 

0.1595231 

0.0176639 

0.124793 

0.1942532 

<.0001* 

Scenario_6 

Baseline.Scenario 

0.1537538 

0.0176639 

0.119024 

0.1884840 

<0001* 

Scenario_6 

Scenaho_5 

0.0966615 

0.0176639 

0.061931 

0.1313917 

<.0001* 

Scenario_3 

Scenario_2 

0.0949846 

0.0176639 

0.060254 

0.1297148 

<0001* 

Scenario_6 

$cenario_4 

0.0898308 

0.0176639 

0.055101 

0.1245609 

<.0001* 

Scenario_3 

Baseline_Scenano 

0.0892154 

0.0176639 

0.054485 

0.1239455 

<.0001* 

Scenario_4 

Sc  enario_2 

0.0696923 

0.0176639 

0.034962 

0.1044225 

<0001* 

Scenario.6 

Scenario_3 

0.0645385 

0.0176639 

0.029808 

0.0992686 

00003* 

Scenano_4 

Baseline_Scenario 

0.0639231 

0.0176639 

0.029193 

0.0986532 

0.0003* 

Scenano_5 

Scenario_2 

0.0628615 

0.0176639 

0.028131 

0.0975917 

00004* 

Scenario_5 

Baseline.Scenario 

0.0570923 

0.0176639 

0.022362 

0.0918225 

0.0013* 

Scenario_3 

Scenario_5 

0.0321231 

0.0176639 

-0.002607 

0.0668532 

0.0698 

Scenario_3 

Scenario_4 

0.0252923 

0.0176639 

-0.009438 

0.0600225 

0.1530 

Scenario_4 

Scenario_5 

0.0068308 

0.0176639 

-0.027899 

0.0415609 

0.6992 

Basel  ine_Scenario  Scenano_2 

0.0057692 

0.0176639 

-0.028961 

0.0404994 

0.7441 

Figure  24.  Comparison  of  each  pair  for  the  proportion  of  successful 
classification  using  Student’s  f-test. 
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The  Connecting  Letters  Report  is  the  simple  way  to  analyze  the  differences 
between  group  means.  The  highest  group  mean  is  always  shown  on  the  top.  Scenario  Six 
has  the  highest  group  mean  amongst  the  other  scenarios — this  is  where  the  largest 
number  of  sensors  are  employed  in  the  ASW  screen  formation.  Scenario  Three,  Scenario 
Four,  and  Scenario  Five  form  the  first  group,  and  Baseline  Scenario  and  Scenario  Two 
form  the  second  group  that  share  the  same  letter  in  the  report.  From  these  results,  we  can 
say  that  the  scenarios  that  have  the  same  number  of  sensors  are  not  considered  as 
significantly  different. 

b.  Time  to  Classify  the  Submarine 

Figure  25  shows  the  visual  comparison  of  the  scenario  means  for  the  time  to 
classify  measure  of  effectiveness.  By  looking  at  the  comparison  circles,  one  can  see  that 
Baseline  Scenario  and  Scenario  Four,  Scenario  Two  and  Scenario  Five,  and  Scenario 
Three  and  Scenario  Six  do  not  display  a  significantly  different  group  means.  If  time  is  an 
important  factor  on  the  mission,  the  ASW  screen  planning  method  should  be  considered 
an  important  factor. 


Figure  25.  The  visual  comparison  of  the  scenario  means  in  terms  of  the  time  to 

classify. 


58 


From  the  detailed  results,  as  shown  in  Figure  26,  one  can  say  that  there  are  three 
comparisons  among  all  pairwise  comparisons  where  no  statistically  significant  difference 
is  found: 

•  Baseline  Scenario  and  Scenario  Four,  where  the  frigates  are  employed  in 
the  inner  screen  and  the  helicopters  in  the  outer  screen. 

•  Scenario  Five  and  Scenario  Two,  where  the  frigates  are  employed  in  the 
inner  screen  and  the  USVs  in  the  intermediate  screen. 

•  Scenario  Three  and  Scenario  Six,  where  the  frigates  are  employed  in 
inner  screen,  the  USVs  in  the  intermediate  screen,  and  the  helicopters  in 
the  outer  screen. 
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Figure  26.  Comparison  of  each  pair  for  time  to  classify  using  Student’s  /-test. 


59 


The  other  pairwise  comparisons  show  that  there  is  a  statistically  significant 
difference.  A  detailed  report  that  compares  each  pair  is  provided  in  Appendix  D. 

6.  Regression  Analysis 

Regression  analysis  is  used  in  simulation  analysis  for  quantifying  the  relationships 
among  variables.  Multiple  linear  regression  is  used  in  this  analysis. 

a.  Multiple  Linear  Regression 

Multiple  linear  regression  is  used  to  model  the  relationship  between  two  or  more 
explanatory  variables  and  a  response  variable  by  fitting  a  linear  equation  to  observed 
data.  The  mean  response  is  modeled  as  a  function  of  multiple  variables.  A  multiple  linear 
regression  with  p  explanatory  variables  has  an  equation  of  the  fomiy  =  /?0  +  fi\X1  + 
/?2x2  +  . . .  +  PpXp,  where  xlt  x2  ...  xp  are  the  explanatory  variables  and  y  is  the  response 
variable. 

MANA  uses  a  random  number  generator  to  randomize  many  properties  in  the 
scenarios.  Therefore,  the  scenarios  can  produce  different  results  for  any  design  point  each 
time  they  are  run.  The  mission  success  is  the  response  variable  to  conduct  a  multiple 
linear  regression.  Since  the  response  variable  has  two  levels,  it  is  hard  to  fit  a  linear 
regression  model.  Another  way  to  fit  a  linear  regression  is  to  summarize  the  data  by 
calculating  the  means  of  each  input  combination.  Therefore,  a  probability  of  mission 
success  is  produced  for  each  design  point.  This  new  data  table  consists  of  390  rows  and  a 
new  response  variable  named  Mean(Mission  Success),  which  is  a  continuous  variable  that 
ranges  from  zero  to  one.  By  fitting  a  linear  regression  model,  we  can  predict  the 
probability  that  the  response  is  equal  to  one  (success). 

In  this  new  table,  the  data  points  for  Scenario  Two  and  Scenario  Four  are 
excluded.  We  assume  that  the  frigates  and  helicopters  have  already  employed  in  all  the 
scenarios.  The  number  of  frigates  and  USV  presence  are  considered  as  categorical  factors 
to  explore  their  effectiveness  on  the  response. 
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We  examine  the  distribution  of  the  mean  of  the  response  using  the  distribution 
platform  in  JMP.  Figure  27  shows  that  the  mean  of  the  response  is  approximately 
normally  distributed  with  a  mean  of  0.461  and  standard  deviation  of  0.1 14. 
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Figure  27.  Distribution  for  the  mean  response. 


b.  Main  Effects  Model 

In  order  to  understand  the  relationship  between  the  input  factors  and  response,  a 
model  is  fitted  using  only  the  main  factors  without  any  interactions.  We  look  at  the  actual 
by  predicted  plot  to  evaluate  the  goodness  of  fit  of  the  model.  From  Figure  28,  the  actual 
by  predicted  plot  shows  that  the  model  fits  the  data  quite  well.  In  this  model,  the  R- 
squared  value  is  around  0.91.  The  R-squared  value  is  a  statistical  measure  which 
represents  how  well  the  regression  line  approximates  the  data  points.  This  model  explains 
91%  of  the  variance  of  the  data. 
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Figure  28.  Actual  by  predicted  plot  and  the  summary  of  the  fit  for  the  main 

effects  model. 


Is  this  main  effect  model  the  correct  model  to  capture  the  relationship  between  the 
input  factors  and  response  variable?  To  answer  this  question,  we  examined  four  different 
assumptions  related  to  the  residuals.  A  residual  value  represents  the  distance  between  the 
observed  value  and  the  fitted  value  in  the  model.  A  graphical  representation  is  an 
effective  way  to  evaluate  the  adequacy  of  the  model.  Figure  29  displays  the  distribution 
of  the  residuals  with  graphs,  quantiles,  and  summary  statistics.  From  Figure  29,  we  see 
two  assumptions  are  satisfied:  the  residuals  are  approximately  normally  distributed,  and 
the  mean  of  the  residuals  is  approximately  equal  to  zero.  A  normal  Q-Q  plot  is  also  used 
to  assess  the  normality.  We  can  see  that  the  approximate  linearity  of  the  points  on  this 
plot  indicates  that  the  residuals  are  normally  distributed. 
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Distributions 
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The  mean  of  the 
residuals  are 
equal  to  zero 
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Figure  29.  Distribution  of  the  residuals  for  the  main  effects  model. 


For  fitting  a  valid  model,  the  error  terms  must  be  uncorrelated  and  have  constant 
variance.  To  check  these  assumptions,  we  created  the  residual  by  predicted  plot  shown  in 
Figure  30.  We  see  that  the  residuals  are  scattered  randomly  about  zero  and  they  have 
constant  variance.  The  assumption  of  uncorrelated  errors  is  also  satisfied  because  there  is 
no  evidence  of  sequencing  of  points. 
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Residual  by  Predicted  Plot 


-n 


0.10 


s 

C* 

I 


“  0.05 
1/> 


» 


3  0.00 


c 

o 


c 

ro 

0/ 


5  -0.10 


02  03  04  0.5  0.6  0.7  03 

Mean(Mission  Success)  Predicted 


Figure  30.  Residual  by  predicted  plot  for  the  main  effects  model. 


The  sorted  parameter  estimates  report  is  useful  in  screening  situations.  This  report 
shows  the  estimates  of  the  parameters  and  conducts  a  hypothesis  test  for  each  model 
parameter  to  test  the  claim  that  the  parameter  estimate  is  equal  to  zero.  In  Figure  31,  the 
parameter  estimates  are  sorted  according  to  their  significance  level.  The  most  significant 
effects  can  be  seen  at  the  top  of  the  report.  There  are  13  highly  significant  factors,  one 
significant  factor,  and  four  insignificant  factors.  The  most  statistically  significant  factor  is 
Frigate  Sonar  Classification  Range,  which  represents  the  reality  in  an  ASW  screen 
formation.  CSV  Presence  is  the  second  one  that  highly  affects  the  response.  All  USV- 
related  factors  are  marked  in  Figure  31.  Among  these  factors,  CSV  Speed  is  the  only 
statistically  insignificant  factor  in  the  model.  The  value  of  each  estimate  has  a  direct 
interpretation  on  the  response.  For  example,  the  presence  of  a  USV  increases  the 
probability  of  mission  success  by  0.089. 
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Sorted  Parameter  Estimates 
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Figure  3 1 .  The  sorted  parameter  estimates  for  the  main  effects  model. 


JMP  produces  a  prediction  expression  which  shows  the  equation  used  to  predict 
the  response  (see  Figure  32).  This  expression  can  be  very  useful  in  the  decision-making 
process. 
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Figure  32.  Prediction  expression  for  the  main  effects  model. 
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c.  Second  Order  Model 


In  this  section,  a  second  order  regression  model  is  developed  over  the  data  set.  We 
included  main  effects,  two-way  interactions,  and  second-order  polynomial  terms  in  the 


model.  A  stepwise  regression  technique  is  used  to  select  a  subset  of  effects  to  fit  a  better 


model.  When  the  additional  terms  are  added  to  the  model,  the  R-squared  value  will 
increase.  It  is  good  to  have  a  higher  R-squared  value,  but  it  is  also  desirable  to  have  fewer 
tenns  in  the  model  to  avoid  overfitting  the  data.  In  brief,  we  are  trying  to  fit  a  valid 
parsimonious  model.  Therefore,  we  created  a  table  using  the  stepwise  regression  step 
history  report  and  then,  we  plotted  R-squared  vs.  the  number  of  terms.  As  Figure  33 
suggests,  after  the  23rd  term,  the  R-squared  value  reaches  a  point  where  adding  more 
terms  will  not  improve  our  model  much. 
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Figure  33.  R-squared  value  increases  with  the  added  terms. 
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The  actual  by  predicted  plot  and  summary  of  fit  for  this  model  is  shown  in  Figure 
34.  The  second  order  model’s  predictions  seem  very  good.  The  R-squared  value  is  around 
0.96.  The  second  order  model  explains  96%  of  the  variance  of  the  output. 
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Figure  34.  Actual  by  predicted  plot  and  the  summary  of  the  fit  for  the  second 

order  model. 


The  R-squared  value  is  really  high,  but  it  does  not  fully  guarantee  that  the  second 
order  model  fits  the  data  well.  We  need  to  check  the  residual  distributions  to  investigate 
how  well  this  model  fits  the  data.  Figure  35  displays  the  distribution  of  the  residuals  with 
graphs,  quantiles,  and  summary  statistics.  This  figure  confirms  that  the  residuals  are 
distributed  around  zero  and  follow  a  normal  distribution.  For  smaller  samples,  JMP 
provides  the  Shapiro-Wilks  test,  which  tests  whether  the  data  comes  from  a  nonnal 
distribution.  Because  the  /7-value  is  greater  than  .05,  we  retain  the  null  hypothesis  that  the 
residuals  come  from  a  normally  distributed  population. 
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Figure  35.  Distribution  of  the  residuals  for  the  second  order  model. 


The  residual  by  predicted  plot  in  Figure  36  indicates  that  the  residuals  have 
constant  variance  and  follow  a  random  pattern.  Therefore,  the  second  order  model 
satisfies  the  assumptions  well. 
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Figure  36.  Residual  by  predicted  plot  for  the  second  order  model. 
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The  sorted  parameter  estimates  are  shown  in  Figure  37.  There  are  15  highly 
significant  factors,  four  significant  factors,  and  four  insignificant  factors.  The  most 
statistically  significant  factor  is  USV  Presence  in  the  second  order  model,  while  Frigate 
Sonar  Classification  Range  is  the  most  statistically  significant  in  the  main  effects  model. 
USV  Speed  factor  is  not  included  in  the  second  order  model  as  distinct  from  the  main 
effects  model. 
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Figure  37.  The  sorted  parameter  estimates  for  the  second  order  model. 


7.  Regression  Tree 

When  the  model  has  non-linearity  and  lots  of  interactions  among  factors,  building 
a  single  regression  model  may  not  be  enough.  An  alternative  technique  for  exploring  the 
effects  of  the  factors  on  the  response  is  building  a  regression  tree.  The  purpose  is  to  fit  a 
model  that  predicts  the  response  variable  based  on  design  factors.  In  regression  tree 
analysis,  the  data  is  recursively  partitioned  into  smaller  regions,  where  the  interactions 
are  easy  to  understand;  then,  a  predictive  model  is  fitted  for  each  cell  of  the  partition.  A 
tree  of  decision  rules  is  formed  until  the  desired  fit  is  obtained.  A  regression  tree  is  a 
useful  technique  because  an  analyst  can  easily  present  the  results  and  insights. 

We  build  a  regression  tree  for  the  probability  of  successful  classification  given  all 

design  factors.  The  data  is  partitioned  into  two  segments  based  on  the  LogWorth  statistic, 

which  is  defined  as  -log10(p- value).  This  statistic  is  reported  in  node  Candidate 
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reports,  as  shown  in  Figure  38.  The  Frigate  Sonar  Classification  Range  column  has  the 
largest  LogWorth,  which  is  noted  by  an  asterisk.  Therefore,  this  factor  defines  the  first 
optimum  split. 
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Figure  38.  Candidates  report  for  the  root  node. 


The  first  five  splits  of  the  regression  tree  are  shown  in  Figure  39.  The 
interpretation  of  the  regression  tree  is  straightforward.  Each  leaf  in  the  decision  tree 
includes  the  probability  of  successful  classification  in  the  Mean  row.  The  first  split  of  the 
partition  tree  occurs  with  the  factor  Frigate  Sonar  Classification  Range,  as  stated  above. 
This  factor  is  the  most  significant  one  in  the  regression  tree  model  as  well  as  in  the  main 
effects  regression  model.  The  original  260  design  points  are  split  into  two  parts:  a  left  leaf 
that  has  68  design  points  and  a  right  leaf  that  has  192  design  points.  If  the  frigate  sonar 
classification  range  is  less  than  or  equal  to  6,343  meters,  the  ASW  screen  has  a  lower 
probability  of  successful  classification.  The  higher  probability  of  successful  classification 
is  evident  when  the  frigate  sonar  classification  range  is  higher  than  or  equal  to  6,343 
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meters.  For  the  left  leaf,  the  next  split  would  happen  on  the  factor  Frigate  Sonar 
Classification  Probability.  For  the  right  leaf,  the  next  split  would  happen  on  the  factor 
USV  Presence  which  is  a  two-level  categorical  variable.  When  USVs  are  not  present  in 
the  model,  the  probability  of  successful  classification  is  0.4461.  When  USVs  are  present 
in  the  model,  the  probability  of  successful  classification  increases  to  0.5246.  The  next 
split  occurs  for  the  USV  Sonar  Classification  Range.  The  probability  of  successful 
classification  is  0.4912  when  the  USV  sonar  classification  range  is  less  than  7,001 
meters,  while  the  probability  of  successful  classification  is  0.5769  when  the  USV  sonar 
classification  range  is  greater  than  or  equal  to  7,001  meters. 
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Figure  39.  The  first  five  splits  of  the  regression  tree.  Colors  and  associated 
means  are  explained  in  the  legend  (located  at  the  top  right). 


We  performed  the  splitting  process  repeatedly  to  find  a  better  R-squared  value. 
But,  this  big  tree  seems  complex,  making  it  difficult  to  display  and  interpret.  Finally,  we 
come  up  with  23  splits  and  observe  an  R-squared  value  of  0.766.  Figure  40  shows  a  plot 
of  R-squared  versus  the  number  of  splits  named  as  split  history.  Figure  41  displays  a 
report  showing  each  factor’s  contribution  to  the  fit  in  the  model. 
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4  Split  History 


Figure  40.  Split  history  for  the  regression  tree  model. 
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Figure  41.  Column  contributions  report  shows  each  factor’s  contribution  to  the 

fit  in  the  model. 
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V.  CONCLUSIONS 


A.  SUMMARY 

This  research  explores  how  USVs  can  complement  and  extend  existing  ASW 
screen  effectiveness  in  detecting  and  classifying  diesel-electric  submarines.  When  an 
HVU  is  screened  by  a  task  force  that  is  conducting  protective  ASW  operations,  the 
submarine  threat  level  can  be  greatly  reduced  with  high  detection  and  classification 
capabilities. 

In  this  study,  the  modeling  first  focuses  on  building  an  existing  ASW  screening 
scenario.  This  baseline  scenario  provides  a  standardized  benchmark  to  evaluate  the  other 
scenarios.  A  generic  scenario  was  built  to  increase  the  understandability.  The  scenarios 
are  implemented  in  the  simulation  modeling  platform  MANA.  We  have  to  state  that  the 
scenarios  built  in  MANA  may  not  necessarily  represent  the  real  ASW  operations  and  the 
assumptions  we  made  about  the  detection  and  classification  of  the  submarine  may  not  be 
necessarily  true.  Thus,  this  simulation  study  cannot  answer  the  detailed  questions,  but  it 
provides  some  useful  insights  about  the  employment  of  USVs  in  ASW  screen  formation. 

B.  ANSWERING  RESEARCH  QUESTIONS 

The  following  research  questions  were  presented  in  Chapter  1 : 

1.  Can  USVs  give  the  same  effectiveness  as  ASW  helicopters  against  diesel- 
electric  submarines  ahead  of  naval  convoys  or  HVUs? 

2.  What  are  the  main  advantages  and  disadvantages  of  employing  USVs  in 
an  ASW  screen  fonnation? 

3.  Which  characteristics  of  USVs  are  the  most  significant  in  ASW? 

4.  How  do  changes  in  decision  parameters  affect  the  probability  of 
classifying  a  diesel-electric  submarine? 

5.  What  strengths  and  drawbacks  does  the  simulation  software  MANA  have 
for  modelling  ASW  scenarios? 

To  answer  the  first  question,  we  conduct  a  comparison  analysis  of  the  scenarios 
where  different  numbers  and  varieties  of  platforms  are  employed  in  an  ASW  screen 
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formation.  In  protective  ASW  operations,  employing  different  platform  and  sensor  types 
can  help  an  ASW  commanders  detect  and  classify  the  stealthy  submarine.  These 
platforms  and  their  sensors  support  and  complement  each  other  to  improve  ASW 
effectiveness.  With  side-by-side  box  plots  and  one-way  analysis  of  the  means  by 
scenarios,  we  find  that  when  the  helicopters  are  replaced  with  USVs,  which  have  the 
same  sensor  type  and  capability,  they  can  provide  the  same  classification  effectiveness  in 
an  ASW  screen  formation.  The  operating  range  of  the  USVs  is  considered  shorter  than 
the  operating  range  of  the  helicopters  because  of  the  autonomy  requirements  of  USVs. 
Therefore,  USVs  are  employed  in  the  intennediate  screen  while  the  helicopters  are 
employed  on  the  outer  screen.  This  gives  the  helicopters  a  great  advantage  against  USVs 
because  the  helicopters  can  extend  the  reach  of  the  frigates  to  the  farther  point  in  the 
ASW  screen  and  provide  an  early  detection  and  classification  of  the  diesel-electric 
submarine. 

Addressing  the  second  question,  we  show  the  primary  advantage  of  employing 
USVs  in  ASW  screen  formation  is  freeing  the  helicopters  to  perform  other  missions,  such 
as  intelligence,  surveillance  and  reconnaissance  (ISR).  The  main  disadvantage  of 
employing  USVs  is  that  they  are  not  nearly  as  efficient  as  the  helicopters  in  early 
detection  and  classification  when  an  early  classification  of  the  enemy  submarine  is 
critical  for  decision  makers.  The  other  disadvantage  would  be  that  USVs  require  a  high 
level  of  autonomy  and  onboard  processing  for  this  mission,  which  means  a  higher  cost 
for  the  development  of  the  dipping  sonar  and  system  design. 

The  proportion  of  successful  classification  is  used  to  measure  the  effectiveness  of 
ASW  screen  formation  in  the  regression  model.  Based  on  this  MOE,  the  most  significant 
characteristic  of  USVs  is  the  classification  range  of  dipping  sonar.  In  ASW,  the 
classification  range  will  depend  on  underwater  conditions,  background  noise  in  the 
ocean,  and  sonar  capability.  The  sonar  parameters  are  mostly  controllable  because  the 
selection  of  the  sonar  type  and  capability  can  be  decided  on  during  the  design  process. 
But,  the  effectiveness  of  sonar  is  limited  by  environmental  conditions.  On  the  other  hand, 
USV  speed  is  viewed  as  an  insignificant  characteristic  in  the  model.  The  reader  must 
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realize  that  US  Vs  are  self-deployed  to  the  intermediate  screen  ahead  of  the  HVU  with 
sufficient  time  and  satisfy  the  requirements  of  station-keeping. 

Many  decision  and  noise  factors  have  a  significant  effect  on  the  response  in  our 
protective  ASW  scenario.  The  sonar  parameters  of  the  frigates  are  really  significant  in  the 
model.  The  first  split  of  the  partition  tree  occurs  with  the  factor  frigate  sonar 
classification  range.  This  factor  has  the  greatest  effect  on  mission  success.  The  number  of 
frigates  is  another  significant  factor  which  affects  the  outcome.  Employing  one  more 
frigate  in  the  screen  will  affect  the  outcome  significantly.  Among  the  noise  factors,  the 
stealthiness  of  the  diesel-electric  submarine  plays  an  important  role  in  the  model,  which 
is  a  significant  factor  in  littoral  ASW  operations.  The  submarine  is  designed  to  submerge 
and  maneuver  quietly  to  avoid  detection.  In  a  noisy  littoral  environment,  the  submarine 
gains  extra  stealth.  The  submarine  can  find  shadow  zones  to  hide  from  active  sonar  and 
approach  an  enemy  without  being  detected. 

Addressing  the  final  research  question,  the  combat  simulation  platfonn  MANA 
has  a  number  of  strengths  to  simulate  maritime  scenarios.  It  is  easy  to  use  and  navigate. 
In  a  maritime  scenario,  the  ships  may  patrol  on  randomly  assigned  routes  in  a  box;  it  is 
straightforward  to  simulate  patrol  boxes  and  random  search  patterns  for  a  specific  agent. 
On  the  other  hand,  MANA  has  several  drawbacks  which  need  to  be  fixed  to  simulate 
maritime  scenarios.  First  of  all,  to  form  an  ASW  screen  effectively,  the  ships  need  to 
know  and  update  target  bearing,  range,  course,  and  speed  in  their  situational  awareness 
maps  at  each  time  step.  Detailed  information  in  an  agent’s  situational  awareness  map  will 
help  the  agent  decide  on  their  next  movements.  Specific  built-in  naval  formation  types 
can  be  added  to  the  next  versions  of  MANA 

The  next  drawback  is  that,  in  an  ASW  scenario,  the  level  of  acoustic  classification 
is  limited  to  two  levels:  submarine  and  non- submarine.  Therefore,  it  is  hard  to  implement 
ASW  contact  classification  procedures.  In  a  realistic  ASW  scenario,  there  are  four  basic 
levels  about  the  certainty  of  classification:  certain  submarine,  probable  submarine, 
possible  submarine,  and  non-submarine.  Classification  procedures  are  important  for 
deciding  on  ASW  force  tactics. 
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C.  FURTHER  RESEARCH 

The  underwater  environment  and  the  thennal  layer  and  their  effects  on  the  sonar 
are  not  simulated  explicitly  in  the  model.  Since  the  littoral  waters  are  complex  and 
chaotic  due  to  several  reasons  that  are  mentioned  in  Chapter  II,  underwater  conditions 
can  have  a  significant  effect  on  the  effectiveness  of  sonar.  For  future  work,  underwater 
conditions  can  be  simulated  to  prove  how  these  conditions  effect  the  detection  and 
classification  of  submarines. 

In  this  study,  we  are  only  interested  in  detecting  and  classifying  the  enemy 
submarine.  The  phases  after  the  classification  phase  are  not  explicitly  addressed  in  this 
study.  USVs  can  contribute  much  more  effectiveness  in  ASW  operations.  They  can  also 
serve  as  armed  escorts  ahead  of  HVUs  with  increased  size  and  payload.  Considering  this 
fact,  localization,  tracking,  and  kill  phases  can  be  simulated  in  future  models. 
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APPENDIX  A.  NOLH  DESIGN  SPREADSHEET 


The  NOLH  design  spreadsheet  is  a  useful  tool  for  designing  large-scale 
simulation  experiments.  In  this  thesis,  the  design  points  are  generated  by  using  the  NOLH 
design  for  16  factors. 
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347 

436 

7 

52 

8401 

8458 

6230 

0.303 

0.453 

0.578 

106 

33 

309 

262 

5625 

1706 

380 

380 

5.6 

60 

8801 

7087 

4087 

0.241 

0.572 

0.466 

117 

27 

214 

183 

5738 

1388 

375 

483 

8 

50 

5715 

8287 

7172 

0.319 

0.541 

0.559 

101 

34 

?77 

267 

5963 

1988 

342 

558 

9.5 

42 

6972 

7944 

8544 

0.213 

0.434 

0.409 

111 

23 

276 

186 

6300 

1594 

567 

427 

5.8 

49 

6743 

8630 

8887 

0.347 

0.419 

0.509 

113 

31 

346 

236 

6750 

2306 

511 

305 

6.3 

46 

5543 

7172 

7601 

0.206 

0.509 

0.438 

108 

20 

183 

326 

6863 

1931 

417 

328 

9 

46 

8572 

5801 

5458 

0.247 

0.503 

0.6 

119 

30 

267 

318 

5400 

1538 

577 

413 

8.4 

40 

7944 

3829 

5972 

0.369 

0.403 

0.494 

101 

20 

354 

276 

6131 

1725 

394 

506 

5.1 

45 

8115 

4687 

4858 

0.209 

0.447 

0.566 

119 

28 

304 

346 

6019 

1275 

413 

544 

6.6 

48 

7429 

5030 

6144 

0.316 

0.584 

0.444 

102 

23 

239 

295 

6806 

1781 

356 

384 

9.8 

59 

6000 

3915 

9058 

0.234 

0.525 

0.547 

111 

28 

256 

340 

6356 

2363 

314 

300 

7.6 

51 

6401 

5287 

6487 

0.334 

0.478 

0.478 

103 

25 

321 

352 

6975 

1463 

441 

581 

5.7 

57 

5486 

6058 

8973 

0.259 

0,516 

0.572 

107 

29 

278 

309 

5906 

2344 

600 

469 

7.3 

56 

6801 

4258 

7430 

0.397 

0.569 

0.488 

114 

27 

250 

203 

7819 

2400 

534 

591 

7.9 

52 

6629 

5201 

5887 

0.328 

0.534 

0.475 

103 

33 

228 

242 

7481 

1744 

548 

539 

6.8 

55 

5886 

3658 

3658 

0.216 

0.472 

0.541 

109 

26 

281 

233 

8213 

2269 

309 

422 

7.7 

56 

7258 

4087 

4772 

0.375 

0.406 

0.503 

104 

31 

335 

191 

7369 

1406 

431 

352 

9.1 

58 

5657 

5972 

4601 

0.25 

0.494 

0.544 

113 

23 

253 

197 

8944 

1650 

305 

511 

6.5 

42 

7029 

4601 

7001 

0.363 

0.6 

0.481 

112 

32 

180 

208 

7256 

2044 

445 

567 

5.9 

47 

8858 

5372 

7687 

0.228 

0.488 

0.597 

120 

26 

323 

225 

8888 

1350 

516 

338 

7.8 

47 

7544 

5115 

6744 

0.309 

0.441 

0.45 

104 

35 

329 

248 

7875 

1838 

473 

366 

10 

41 

8058 

4772 

8373 

0.244 

0.588 

0.584 

105 

24 

197 

287 

8325 

2081 

581 

347 

5.5 

43 

6458 

7258 

4344 

0.2 

0.578 

0.472 

110 

29 

208 

360 

8156 

1425 

492 

408 

6.9 

49 

5600 

8373 

5544 

0.313 

0.409 

0.588 

114 

21 

315 

323 

7313 

2288 

398 

586 

8.8 

45 

6286 

6915 

5287 

0.222 

0.481 

0.425 

116 

33 

293 

329 

8719 

1575 

338 

497 

8.7 

47 

6858 

9058 

4001 

0.344 

0.538 

0.516 

118 

24 

203 

290 

7706 

1444 

422 

323 

6.1 

57 

8687 

6658 

8630 

0.263 

0.544 

0.419 

108 

34 

242 

312 

8606 

2381 

403 

459 

5.3 

56 

7715 

7344 

8287 

0.322 

0.45 

0.531 

106 

22 

307 

281 

8831 

1688 

530 

530 

9.6 

59 

8458 

6572 

7773 

0.219 

0.425 

0.447 

115 

30 

349 

335 

7988 

1969 

572 

502 

8.3 

52 

8515 

7858 

7087 

0.306 

0.556 

0.594 

100 

28 

270 

270 

7200 

1800 

450 

450 

7.5 

50 

7315 

6401 

6401 

0.3 

0.5 

0.5 

91 

34 

295 

301 

8550 

1481 

361 

455 

5.2 

46 

7144 

4001 

8115 

0.275 

0.438 

0.463 

82 

24 

340 

284 

7763 

2100 

436 

375 

6.4 

42 

6343 

6487 

7858 

0.331 

0.531 

0.569 

84 

30 

188 

321 

7931 

1369 

553 

464 

8 

48 

6229 

4344 

6572 

0.297 

0.547 

0.422 

94 

22 

231 

278 

8775 

1894 

520 

520 

9.4 

40 

5829 

5715 

8715 

0.359 

0.428 

0.534 

83 

28 

326 

357 

8663 

2213 

525 

417 

7 

50 

8915 

4515 

5630 

0.281 

0.459 

0.441 

99 

21 

318 

273 

8438 

1613 

558 

342 

5.5 

58 

7658 

4858 

4258 

0.388 

0.566 

0.591 

89 

32 

264 

354 

8100 

2006 

333 

473 

9.2 

51 

7887 

4172 

3915 

0.253 

0.581 

0.491 

88 

24 

194 

304 

7650 

1294 

389 

595 

8.8 

54 

9087 

5630 

5201 

0.394 

0.491 

0.563 

93 

35 

357 

214 

7538 

1669 

483 

572 

6 

54 

6058 

7001 

7344 

0.353 

0.497 

0.4 

81 

25 

273 

222 

9000 

2063 

323 

488 

6.6 

60 

6686 

8973 

6830 

0.231 

0.597 

0.506 

99 

35 

186 

264 

8269 

1875 

506 

394 

9.9 

55 

6515 

8115 

7944 

0.391 

0.553 

0.434 

81 

27 

236 

194 

8381 

2325 

488 

356 

8.4 

53 

7201 

7773 

6658 

0.284 

0.416 

0.556 

98 

32 

301 

245 

7594 

1819 

544 

516 

5.2 

41 

8630 

8887 

3744 

0.366 

0.475 

0.453 

89 

27 

284 

200 

8044 

1238 

586 

600 

7.4 

49 

8230 

7515 

6315 

0.266 

0.522 

0.522 

97 

30 

219 

188 

7425 

2138 

459 

319 

9.3 

43 

9144 

6744 

3829 

0.341 

0.484 

0.428 

93 

26 

262 

231 

8494 

1256 

300 

431 

7.7 

44 

7829 

8544 

5372 

0.203 

0.431 

0.513 

86 

28 

290 

338 

6581 

1200 

366 

309 

7.1 

48 

8001 

7601 

6915 

0.272 

0.466 

0.525 

98 

22 

312 

298 

6919 

1856 

352 

361 

8.2 

45 

8744 

9144 

9144 

0.384 

0.528 

0.459 

91 

29 

259 

307 

6188 

1331 

591 

478 

7.3 

44 

7372 

8715 

8030 

0.225 

0.594 

0.497 

96 

24 

205 

349 

7031 

2194 

469 

548 

5.9 

43 

8973 

6830 

8201 

0.35 

0.506 

0.456 

87 

32 

287 

343 

5456 

1950 

595 

389 

8.5 

58 

7601 

8201 

5801 

0.238 

0.4 

0.519 

88 

23 

360 

332 

7144 

1556 

455 

333 

9.1 

53 

5772 

7430 

5115 

0.372 

0.513 

0.403 

80 

29 

217 

315 

5513 

2250 

384 

563 

7.2 

53 

7086 

7687 

6058 

0.291 

0.559 

0.55 

96 

20 

211 

293 

6525 

1763 

427 

534 

5 

59 

6572 

8030 

4429 

0.356 

0.413 

0.416 

95 

31 

343 

253 

6075 

1519 

319 

553 

9.5 

57 

8172 

5544 

8458 

0.4 

0.422 

0.528 

90 

26 

332 

180 

6244 

2175 

408 

492 

8.1 

51 

9030 

4429 

7258 

0.288 

0.591 

0.413 

86 

34 

225 

217 

7088 

1313 

502 

314 

6.2 

55 

8344 

5887 

7515 

0.378 

0.519 

0.575 

84 

22 

248 

211 

5681 

2025 

563 

403 

6.3 

53 

7772 

3744 

8801 

0.256 

0.463 

0.484 

83 

31 

338 

250 

6694 

2156 

478 

577 

8.9 

43 

5943 

6144 

4172 

0.338 

0.456 

0.581 

92 

21 

298 

228 

5794 

1219 

497 

441 

9.7 

44 

6915 

5458 

4515 

0.278 

0.55 

0.469 

94 

33 

233 

259 

5569 

1913 

370 

370 

5.4 

41 

6172 

6230 

5030 

0.381 

0.575 

0.553 

85 

25 

191 

205 

6413 

1631 

328 

398 

6.7 

48 

6115 

4944 

5715 

0.294 

0.444 

0.406 
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APPENDIX  B.  DISTRIBUTIONS  OF  “STEPS”  COLUMNS  BY 

SCENARIOS 


Distributions  Scenario  =  Bascline.Scenario 
Steps 


Quantiles 

1009% 

maximum 

32390 

99.5% 

3057754 

97.5% 

29265-15 

90.0% 

274432 

75.0% 

quartile 

227295 

50.0% 

median 

15723 

25.0% 

quartile 

8344 

10.0% 

4855 

2.5% 

242095 

0.5% 

1905.77 

0.0% 

minimum 

736 

Summary  Statistics 

Mean 

15839072 

Std  Dev 

81709629 

Std  Err  Mean 

51.72572 

Upper  95%  Mean 

15940458 

lower  95%  Mean 

15737.687 

N 

24953 

Distributions  Scenario = Scenario_4 
Steps 


Distributions  Scenario =Scenario_2 
Steps 


Quantiles 


1000% 

E 

g 

i 

32581 

99.5% 

30416315 

97.5% 

29176525 

90.0% 

274142 

75.0% 

quartile 

2337725 

50.0% 

median 

166065 

25.0% 

quartile 

9461 

10.0% 

5475.7 

2.5% 

2876475 

0.5% 

2034995 

0.0% 

minimum 

635 

Summary  Statistics 


Mean  16476555 

Std  Dev  80049544 

Std  Err  Mean  51.060562 
Upper  95%  Mean  16576937 
lower  95%  Mean  16376.773 
N  24578 

Distributions  Scenario = Scenario_5 
Steps 


Distributions  Scenario =Scenario_3 
Steps 


Quantiles 

1000%  maximum 

32076 

99.5% 

2913247 

97.5% 

27770 

90.0% 

25078 

75.0% 

quartile 

19991.75 

50.0% 

median 

13740 

25.0% 

quartile 

6591 

10.0% 

39663 

2.5% 

159765 

0.5% 

1168.295 

0.0% 

minimum 

614 

Summary  Statistics 

Mean 

13785252 

Std  Dev 

77427812 

Std  Err  Mean 

44.153009 

Upper  95%  Mean 

138716*4 

lower  95%  Mean 

1369861 

N 

30752 

Distributions  Scenario = Scenario_6 
Steps 


Quantiles 


1000% 

maximum 

32576 

99.5% 

3019846 

97.5% 

28952275 

90.0% 

27143 

75.0% 

quartile 

22796 

50.0% 

median 

158175 

25.0% 

quartile 

81595 

10.0% 

4619 

2.5% 

2301 

0.5% 

175728 

0.0% 

minimum 

735 

Summary  Statistics 


Mean 

15768215 

Std  Dev 

8182.1288 

Std  Err  Mean 

47.957897 

Upper  95%  Mean 

15862315 

Lower  95%  Mean 

15674.116 

N 

29108 

Quantiles 


1000% 

maximum 

32250 

99.5% 

3001735 

97.5% 

28900375 

90.0% 

272715 

75.0% 

quartile 

2342075 

50.0% 

median 

16937 

25.0% 

quartile 

95755 

10.0% 

52755 

2.5% 

28965 

0.5% 

160005 

0.0% 

minimum 

677 

Summary  Statistics 


Mean 

1652504 

Std  Dev 

79785985 

Std  Err  Mean 

47.125743 

Upper  95%  Mean 

16617409 

lower  95%  Mean 

16432672 

N 

28664 

Quantiles 


1000% 

maximum 

31361 

99.5% 

2868004 

97.5% 

273253 

90.0% 

24682 

75.0% 

quartile 

19981 

50.0% 

median 

13828 

25.0% 

quartile 

6429 

10.0% 

3583 

2.5% 

1457.7 

0.5% 

1152 

0.0% 

minimum 

m 

Summary  Statistics 


Mean 

136428 

Std  Dev 

77363432 

Std  Err  Mean 

41.382769 

Upper  95%  Mean 

13723011 

Lower  95%  Mean 

13561688 

N 

34047 
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APPENDIX  C.  DETAILED  COMPARISONS  REPORT  FOR  T-TEST 
(MOE1-THE  PROPORTION  OF  SUCCESSFUL  CLASSIFICATION) 


This  detailed  report  provides  the  paired  /-test  comparisons  of  the  scenarios.  The 
statistical  text  includes  the  difference  between  the  levels,  standard  error,  and  confidence 
intervals,  t-ratios,  p-values,  and  degrees  of  freedom.  A  plot  is  also  provided  for  the 
comparison  on  the  right  of  each  report. 


zi  Detailed  Comparisons  Report 


Comparing  Scenario_2  with  Baseline_Scenario 

Difference 

-0.00577  t  Ratio 

-0.32661 

Std  Err  Dif 

0.01766  DF 

384 

Upper  CL  Dif 

0.02896  Prob  >  |t| 

0.7441  A 

Lower  CL  Dif 

-0.04050  Prob  >  t 

0.6279  A 

Confidence 

0.95  Prob  <  t 

0.3721 

-0.06 

-0.02  0.00  0.02  0.04  0.06 

Comparing  Scenario_3  with  Baseline_Scenario 

Difference 

0.089215  t  Ratio 

5.050708 

Std  Err  Dif 

0.017664  DF 

384 

Upper  CL  Dif 

0.123946  Prob  >  |t| 

<■0001* 

Lower  CL  Dif 

0.054485  Prob  >  t 

<.0001* 

Confidence 

0.95  Prob  <  t 

1.0000 

,  |  , 

-0.10 

-0.05 

0.00  0.05 

0.10 

Comparing  Scenario_3  with  Scenario_2 

Difference 

0.094985  t  Ratio 

5.377318 

Std  Err  Dif 

0.017664  DF 

384 

Upper  CL  Dif 

0.129715  Prob  >  |t| 

<.0001* 

Lower  CL  Dif 

0.060254  Prob  >  t 

<.0001* 

Confidence 

0.95  Prob  <  t 

1.0000 

,  1, 

-0.10 

-0.05 

0.00  0.05 

0.10 

Comparing  Scenario_4  with  Baseline_Scenario 

Difference 

0.063923  t  Ratio 

3.6  1  8846 

Std  Err  Dif 

0.017664  DF 

384 

Upper  CL  Dif 

0.098653  Prob  >  |t| 

0.0003’ 

Lower  CL  Dif 

0.029193  Prob  >  t 

0.0002* 

Confidence 

0.95  Prob  <  t 

0.9998 

,1 

-0.08 

-0.04 

0.00  0.04 

0.08 

Comparing  Scenario_4  with  Scenario_2 

Difference 

0.069692  t  Ratio 

3.945457 

Std  Err  Dif 

0.017664  DF 

384 

Upper  CL  Dif 

0.104422  Prob  >  |t| 

<.0001* 

Lower  CL  Dif 

0.034962  Prob  >  t 

<.0001* 

Confidence 

0.95  Prob  <  t 

1.0000 

-0.08  -0.04  0.00  0.04  0.08 
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Comparing  Scenario_4  with  Scenario_3 

Difference 

Std  Err  Dif 
Upper  CL  Dif 
Lower  CL  Dif 
Confidence 

-0.02529  t  Ratio 
0.01766  DF 

0.00944  Prob  >  |t| 
-0.06002  Prob  >  t 
0.95  Prob  <  t 

-1.43186 

384 

0.1530 

0.9235  A 

0.0765 

\ 

-0.06 


-0.02  0.00  0.02  0.04  0.06 


Comparing  Scenario_5  with  Baseline_Scenario 

Difference 

0.057092  t  Ratio 

3.232139 

ps. 

Std  Err  Dif 

0.017664  DF 

384  / 

\ 

\ 

Upper  CL  Dif 

0.091822  Prob  >  |t| 

0.0013*  / 

Lower  CL  Dif 

0.022362  Prob  >  t 

0.0007*  / 

Confidence 

0.95  Prob  <  t 

0.9993 

M 

T - 1 - 1 - 1 - - ' - 1 - ' - 1 — 1 1 


-0.06  -0.02  0.00  0.02  0.04  0.06 


Comparing  Scenario_5  with  Scenario_2 

Difference 

0.062862  t  Ratio 

3.55875 

Std  Err  Dif 

0.017664  DF 

384  / 

Upper  CL  Dif 

0.097592  Prob  >  |t| 

0.0004*  / 

Lower  CL  Dif 

0.028131  Prob  >  t 

0.0002*  / 

Confidence 

0.95  Prob  <  t 

0.9998 

- 1 - - 1 - — r — i 

-0.08  -0.04  0.00  0.04  0.08 


Comparing  Scenario_5  with  Scenario_3 

Difference 

-0.03212  t  Ratio 

-1.81857 

l-v 

Std  Err  Dif 

0.01766  DF 

384  / 

Upper  CL  Dif 

0.00261  Prob  >  |t| 

0.0698 

Lower  CL  Dif 

-0.06685  Prob  >  t 

0.9651 

Confidence 

0.95  Prob  <  t 

0.0349*  -A 

V- 

-0.06  -0.02  0.00  0.02  0.04  0.06 


A  Comparing  Scenario_5  with  Scenario_4 


Difference  -0.00683  t  Ratio  -0.38671 

Std  Err  Dif  0.01766  DF  384 

Upper  CL  Dif  0.02790  Prob  >  |t|  0.6992 

Lower  CL  Dif  -0.04156  Prob  >  t  0.6504 

Confidence  0.95  Prob  <  t  0.3496 


-0.06 


-0.02  0.00  0.02  0.04  0.06 


a  Comparing  Scenario_6  with  Baseline_Scenario 


Difference 
Std  Err  Dif 
Upper  CL  Dif 
Lower  CL  Dif 
Confidence 


0.153754  t  Ratio 
0.017664  DF 
0.188484  Prob  >  |t| 
0.119024  Prob  >  t 
0.95  Prob  <  t 
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A  Comparing  Scenario_6  with  Scenario_2 

0.159523 
0.017664 
0.194253 
0.124793 
0.95 

-0.20  -0.10  0.00  0.10  0.20 

A  Comparing  Scenario_6  with  Scenario_3 

0.064538 
0.017664 
0.099269 
0.029808 
0.95 

-0.08  -0.04  0.00  0.04  0.08 


t  Ratio 

3.653685 

/ 

DF 

384 

Prob  >  |t| 

0.0003* 

Prob  >  t 

0.0001* 

Prob  <  t 

0.9999 

Difference 
Std  Err  Dif 
Upper  CL  Dif 
Lower  CL  Dif 
Confidence 


t  Ratio 

9.031003 

DF 

384  / 

Prob  >  |t| 

<.0001*  / 

Prob  >  t 

<.0001*  / 

Prob  <  t 

1.0000 

Difference 
Std  Err  Dif 
Upper  CL  Dif 
Lower  CL  Dif 
Confidence 


A  Comparing  Scenario_6  with  Scenario_4 


Difference 

Std  Err  Dif 
Upper  CL  Dif 
Lower  CL  Dif 
Confidence 

0.089831 

0.017664 

0.124561 

0.055101 

0.95 

t  Ratio 

DF 

Prob  >  |t| 
Prob  >  t 
Prob  <  t 

5.085546 

384 

<.0001* 

<.0001* 

1.0000 

/ 

V  l 

-0.10 

-0.05  0.< 

X)  0.05  0.10 

Comparing  Scenario_6  with  Scenario_5 

Difference 

0.096662 

t  Ratio 

5.472253 

( 

/ 

IN 

Std  Err  Dif 

0.017664 

DF 

384 

\ 

Upper  CL  Dif 

0.131392 

Prob  >  |t| 

<•0001* 

\ 

Lower  CL  Dif 

0.061931 

Prob  >  t 

<.0001* 

/ 

J 

Confidence 

0.95 

Prob  <  t 

1.0000 

V  1 

-0.10  -0.05  0.00  0.05  0.10 
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APPENDIX  D.  DETAILED  COMPARISONS  REPORT  FOR  T-TEST 
(MOE2-TIME  TO  CLASSIFY  THE  SUBMARINE) 


A  Detailed  Comparisons  Report 

A 

Comparing  Scenario_2  with  Baseline_Scenario 

Difference 

642.391  t  Ratio  3.967209 

Std  Err  Dif 

161.925  DF  384  / 

Upper  CL  Dif 

960.763  Prob  >  |t|  <.0001*  / 

Lower  CL  Dif 

324.020  Prob  >  t  <.0001*  / 

Confidence 

0.95  Prob  <  t  1.0000 

.  V  ,1 

-800  -400  0  200  600 


Comparing  Scenario_3  with  Baseline_Scenario 

Difference 

-2063.9  t  Ratio 

-12.7461 

Std  Err  Dif 

161.9  DF 

384 

Upper  CL  Dif 

-1745.5  Prob  >  |t| 

<.0001* 

Lower  CL  Dif 

-2382.3  Prob  >  t 

1.0000 

Confidence 

0.95  Prob  <  t 

<.0001* 

-2000  -1000  0  1000  2000 


4  Comparing  Scenario_3  with  Scenario_2 


Difference 

-27063  t  Ratio 

-16.7133  a 

Std  Err  Dif 

161.9  DF 

384 

Upper  CL  Dif 

-2387.9  Prob  >  |t| 

<.0001*  / 

Lower  CL  Dif 

-3024.7  Prob  >  t 

1.0000 

Confidence 

0.95  Prob  <  t 

<.0001* 

-3000  -1000  0  1000  2000 


41  Comparing  Scenario_4  with  Baseline_Scenario 


Difference 
Std  Err  Dif 
Upper  CL  Dif 
Lower  CL  Dif 
Confidence 


-57.80 

t  Ratio 

-0.35698 

161.93 

DF 

384 

260.57 

Prob  > 

|t| 

0.7213 

-376.18 

Prob  > 

t 

0.6393 

0.95 

Prob  < 

t 

0.3607 

-400  -200  0  200  400 


Comparing  Scenario_4  with  Scenario_2 

Difference 

-7002  t  Ratio 

-4.32419 

Std  Err  Dif 

161.9  DF 

384 

Upper  CL  Dif 

-381.8  Prob  >  |t| 

<.0001* 

Lower  CL  Dif 

-1018.6  Prob  >  t 

1.0000 

Confidence 

0.95  Prob  <  t 

<.0001* 

-800  -400  0  200  600 
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A  Comparing  Scenario_4  with  Scenario_3 


Difference 

2006.11  t  Ratio 

12.38911 

Std  Err  Dif 

161.93  DF 

384 

Upper  CL  Dif 

232448  Prob  >  |t| 

<.0001*  / 

Lower  CL  Dif 

1687.74  Prob  >  t 

<.0001*  / 

Confidence 

0.95  Prob  <  t 

1.0000 

\  | 

-2000  -1000  C 

1000  2000 

Comparing  Scenario_5  with  Baseline_Scenario 

Difference 

670.553  t  Ratio 

4.141124 

IN 

Std  Err  Dif 

161.925  DF 

384  / 

\ 

\ 

Upper  CL  Dif 

988.924  Prob  >  |t| 

<.0001*  / 

\ 

\ 

\  1 

Lower  CL  Dif 

352.181  Prob  >  t 

<.0001*  / 

Confidence 

0.95  Prob  <  t 

1.0000 

1 

-800  -400 


o  200 


600 


a  Comparing  Scenario_5  with  Scenario_2 


Difference 
Std  Err  Dif 
Upper  CL  Dif 
Lower  CL  Dif 
Confidence 


28.16  t  Ratio 
161.93  DF 
346.53  Prob  >  |t| 
-29021  Prob  >  t 
0.95  Prob  <  t 


0.173915 

384 

0.8620 

0.4310 

0.5690 


-400  -200 


200  400 


Comparing  Scenario 5  with  Scenario 3 

Difference 

273447  t  Ratio 

16.88722  i 

Std  Err  Dif 

161.93  DF 

384 

Upper  CL  Dif 

3052.84  Prob  >  |t| 

<.oooi*  / 

Lower  CL  Dif 

2416.10  Prob  >  t 

<.0001*  / 

Confidence 

0.95  Prob  <  t 

1.0000 

A_ L 

|  1 

-3000 

-1000  C 

1000  2000 

Comparing  Scenario_5  with  Scenario_4 

Difference 

728.36  t  Ratio 

4.498107 

/ 

\ 

\ 

Std  Err  Dif 

161.93  DF 

384 

/ 

Upper  CL  Dif 

1046.73  Prob  >  |t| 

<.0001* 

\ 

\ 

Lower  CL  Dif 

409.99  Prob  >  t 

<.0001* 

Confidence 

0.95  Prob  <  t 

1.0000 

V,  1 

-1000 

'"T  1 

-500  C 

I  1  al  1 

500  1000 

Comparing  Scenario_6  with  Baseline_Scenario 

Difference 

-2201.6  t  Ratio 

-13.5963 

Std  Err  Dif 

161.9  DF 

384 

Upper  CL  Dif 

-18832  Prob  >  |t| 

<.0001* 

Lower  CL  Dif 

-2520.0  Prob  >  t 

1.0000 

Confidence 

0.95  Prob  <  t 

<.0001* 

-2000  -1000  0  1000  2000 
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a  Comparing  Scenario_6  with  Scenario_2 


Difference 

-2844.0  t  Ratio 

-17.5636 

I 

Std  Err  Dif 

161.9  DF 

384 

/ 

1 

Upper  CL  Dif 

-2525.6  Prob  >  |t| 

<.ooor 

Lower  CL  Dif 

-31624  Prob  >  t 

1.0000 

/ 

\ 

Confidence 

0.95  Prob  <  t 

<.0001* 

-3000  -1000  C 

1000  2000 

Comparing  Scenario 6  with  Scenario 3 

Difference 

-137.68  t  Ratio 

-0.85025 

Std  Err  Dif 

161.93  DF 

384 

Upper  CL  Dif 

180.69  Prob  >  |t| 

0.3957  A 

Lower  CL  Dif 

-456.05  Prob  >  t 

0.8021 

Confidence 

0.95  Prob  <  t 

0.1979 

-400  -200  0  200  400 


41  Comparing  Scenario_6  with  Scenario_4 


Difference 

-2143.8  t  Ratio 

-13.2394 

Std  Err  Dif 

161.9  DF 

384 

\ 

Upper  CL  Dif 

-18254  Prob  >  |t| 

<.0001* 

l 

Lower  CL  Dif 

-24622  Prob  >  t 

1.0000 

Confidence 

0.95  Prob  <  t 

<.0001* 

-2000  -1000  C 

1000  2000 

Comparing  Scenario 6  with  Scenario 5 

Difference 

-2872.1  t  Ratio 

-17.7375 

Std  Err  Dif 

161.9  DF 

384 

Upper  CL  Dif 

-2553.8  Prob  >  |t| 

<.0001*  1 

Lower  CL  Dif 

-3190.5  Prob  >  t 

1.0000 

1 

Confidence 

0.95  Prob  <  t 

<.0001* 

-4000  -2000  0  1000  3000 
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